bfweb.co.uk
robots.txt

Robots Exclusion Standard data for bfweb.co.uk

Resource Scan

Scan Details

Site Domain bfweb.co.uk
Base Domain bfweb.co.uk
Scan Status Ok
Last Scan2025-05-07T19:52:56+00:00
Next Scan 2025-06-06T19:52:56+00:00

Last Scan

Scanned2025-05-07T19:52:56+00:00
URL https://bfweb.co.uk/robots.txt
Domain IPs 109.109.227.178
Response IP 109.109.227.178
Found Yes
Hash c2c6244d131803919756f24f4295a379507907c0a06d805cdb7d4ef4fefb1833
SimHash 21555c82d410

Groups

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

googlebot-mobile

Rule Path
Disallow

mediapartners-google

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

msnbot

Rule Path
Disallow

slurp

Rule Path
Disallow

teoma

Rule Path
Disallow

gigabot

Rule Path
Disallow

nutch

Rule Path
Disallow

ia_archiver

Rule Path
Disallow

yahoo-mmcrawler

Rule Path
Disallow

asterias

Rule Path
Disallow

yahoo-blogs/v3.9

Rule Path
Disallow

scrubby

Rule Path
Disallow

robozilla

Rule Path
Disallow

mozilla/2.0

Rule Path
Disallow

askjeeves

Rule Path
Disallow

jeeves

Rule Path
Disallow

fast-webcrawler

Rule Path
Disallow

architextspider

Rule Path
Disallow

lycos

Rule Path
Disallow

uk searcher spider

Rule Path
Disallow

inktomi slurp

Rule Path
Disallow

inktomi

Rule Path
Disallow

*

Rule Path
Disallow

Other Records

Field Value
sitemap http://www.bfweb.co.uk/sitemap.xml.gz