Log In Sign Up

grcade.co.uk
robots.txt

Robots Exclusion Standard data for grcade.co.uk

Resource Scan

Scan Details

Site Domain grcade.co.uk
Base Domain grcade.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-03T14:53:50+00:00
Next Scan 2026-01-01T14:53:50+00:00

Last Successful Scan

Scanned2025-06-06T09:28:59+00:00
URL https://grcade.co.uk/robots.txt
Domain IPs 94.23.1.68
Response IP 94.23.1.68
Found Yes
Hash 652eed06ece29a8b1431995171b5d08c0dd98dc005e84ef84161224c1f94afa8
SimHash 3e178911c6f5

Groups

*

Rule Path
Disallow

anthropicai
openai
sogou
ahrefsbot
semrushbot
ia_archiver
ai2bot
ai2bot-dolma
amazonbot
anthropic-ai
applebot
applebot-extended
bytespider
ccbot
chatgpt-user
claude-web
claudebot
cohere-ai
cohere-training-data-crawler
crawlspace
diffbot
duckassistbot
facebookbot
friendlycrawler
google-extended
googleother
googleother-image
googleother-video
gptbot
iaskspider/2.0
icc-crawler
imagesiftbot
img2dataset
isscyberriskcrawler
kangaroo bot
meta-externalagent
meta-externalfetcher
oai-searchbot
omgili
omgilibot
pangubot
perplexitybot
petalbot
scrapy
semrushbot-ocob
semrushbot-swa
sidetrade indexer bot
timpibot
velenpublicwebcrawler
webzio-extended
youbot

Rule Path
Disallow /
Back to top

Comments

  • The following two lines are saying: all user agents are allowed to crawl all of the site
  • Another way of saying the same thing could be:
  • User-agent: *
  • Allow: /
  • You don't really need any of these, if you don't put anything the implicit message is the same: allow all crwalers to the full site
  • The following lines are listing all known (to the date of this file) AI crawlers
  • The last line is saying: all of the above bots are not allowed to crawl any part of the site
Back to top