brewmaster.org.uk
robots.txt

Robots Exclusion Standard data for brewmaster.org.uk

Resource Scan

Scan Details

Site Domain brewmaster.org.uk
Base Domain brewmaster.org.uk
Scan Status Ok
Last Scan2025-09-17T12:56:11+00:00
Next Scan 2025-10-17T12:56:11+00:00

Last Scan

Scanned2025-09-17T12:56:11+00:00
URL https://brewmaster.org.uk/robots.txt
Redirect https://www.brewmaster.org.uk/robots.txt
Redirect Domain www.brewmaster.org.uk
Redirect Base brewmaster.org.uk
Domain IPs 185.151.30.148, 2a07:7800::148
Redirect IPs 185.151.30.148, 2a07:7800::148
Response IP 185.151.30.148
Found Yes
Hash 65a4fb3e642e06c31a05c9ce289f2dc2d1690444ac03f24671ede46293223761
SimHash 6042d0175490

Groups

googlebot

Rule Path
Disallow /_notes/
Disallow /cp/
Disallow /images/
Disallow /img/
Disallow /phpmailer/

Other Records

Field Value
crawl-delay 20

bingbot

Rule Path
Disallow /_notes/
Disallow /cp/
Disallow /images/
Disallow /img/
Disallow /phpmailer/

Other Records

Field Value
crawl-delay 20

teoma

Product Comment
teoma (Ask/Teoma)
Rule Path
Disallow /_notes/
Disallow /cp/
Disallow /images/
Disallow /img/
Disallow /phpmailer/

Other Records

Field Value
crawl-delay 20

facebookexternalhit

Rule Path
Disallow /_notes/
Disallow /cp/
Disallow /images/
Disallow /img/
Disallow /phpmailer/

Other Records

Field Value
crawl-delay 20

ia_archiver

Rule Path
Disallow /_notes/
Disallow /cp/
Disallow /phpmailer/

Other Records

Field Value
crawl-delay 20

tweetmemebot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow
Allow /img/

*

Rule Path
Disallow /

googlebot-image

Rule Path
Disallow /

irlbot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

webalta crawler

Rule Path
Disallow /

yandex

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

Comments

  • robots.txt originally generated at http://www.mcanerin.com
  • These are not needed but keep for reference