Log In Sign Up

locksmiths.co.uk
robots.txt

Robots Exclusion Standard data for locksmiths.co.uk

Resource Scan

Scan Details

Site Domain locksmiths.co.uk
Base Domain locksmiths.co.uk
Scan Status Ok
Last Scan2024-09-20T05:30:04+00:00
Next Scan 2024-10-20T05:30:04+00:00

Last Scan

Scanned2024-09-20T05:30:04+00:00
URL https://locksmiths.co.uk/robots.txt
Redirect https://www.locksmiths.co.uk/robots.txt
Redirect Domain www.locksmiths.co.uk
Redirect Base locksmiths.co.uk
Domain IPs 104.26.2.234, 104.26.3.234, 172.67.72.19, 2606:4700:20::681a:2ea, 2606:4700:20::681a:3ea, 2606:4700:20::ac43:4813
Redirect IPs 104.26.2.234, 104.26.3.234, 172.67.72.19, 2606:4700:20::681a:2ea, 2606:4700:20::681a:3ea, 2606:4700:20::ac43:4813
Response IP 104.26.3.234
Found Yes
Hash 465c80d50423089fd0bb0fda322b9ccb10a346d9ff27346fb0520841637cf811
SimHash d6345d59c7b5

Groups

googlebot

Rule Path
Allow /
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

bingbot

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

twitterbot

Rule Path
Allow /

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 3

yahoo

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

applebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

bleriot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

firstra

Rule Path
Disallow /

wow64

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

trendsmapresolver

Rule Path
Disallow /

fast

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

mail.ru_bot

Rule Path Comment
Disallow / blocks access to the entire site

wesee

Rule Path
Disallow /

piplbot

Rule Path
Disallow /
Back to top

Comments

  • CRAWL LIMITED BOTS
  • French Search Engine called Qwant
  • BLOCKED BOTS
  • Baiduspider
  • Yandex
  • Copied from wikipedia robots.txt
  • Crawlers that are kind enough to obey, but which we'd rather not have
  • unless they're feeding search engines.
  • Some bots are known to be trouble, particularly those designed to copy
  • entire sites. Please obey robots.txt.
  • Misbehaving: requests much too fast:
  • The 'grub' distributed client has been *very* poorly behaved.
  • Doesn't follow robots.txt anyway, but...
  • Hits many times per second, not acceptable
  • http://www.nameprotect.com/botinfo.html
  • A capture bot, downloads gazillions of pages with no public benefit
  • http://www.webreaper.net/
Back to top