michaelchell.co.uk
robots.txt

Robots Exclusion Standard data for michaelchell.co.uk

Resource Scan

Scan Details

Site Domain michaelchell.co.uk
Base Domain michaelchell.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-07-07T00:47:16+00:00
Next Scan 2025-09-05T00:47:16+00:00

Last Successful Scan

Scanned2025-04-16T00:16:55+00:00
URL https://www.michaelchell.co.uk/robots.txt
Domain IPs 18.203.114.168, 52.48.108.215, 54.194.51.130
Response IP 52.48.108.215
Found Yes
Hash 9d48d011f39889dae443f702e60ae1a7934349a6df1c55be3dbb460ca278aaca
SimHash 71248943c690

Groups

*

Rule Path
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /pkginfo/
Disallow /var/
Disallow /setup/
Disallow /pub/errors/
Disallow /pub/static/
Disallow /pub/media/
Disallow /generated/
Disallow /admin/
Disallow /customer/
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /cart/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /customer/account/create/
Disallow /wishlist/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalogsearch/
Disallow /search/
Disallow /*?dir=*
Disallow /*?limit=*
Disallow /*?mode=*
Disallow /*?order=*
Disallow /*?price=*
Disallow /*?cat=*
Disallow /*?q=*
Disallow /*?*retailstore*
Disallow /*?SID=
Disallow /*?___from_store=
Disallow /*?___store=
Disallow /*?___currency=
Allow /pub/media/
Allow /pub/static/
Allow /static/frontend/
Allow /media/catalog/
Allow /skin/frontend/

ahrefsbot
ahrefsbot/7.0
semrushbot/7~bl
ai2bot
ai2bot-dolma
amazonbot
anthropic-ai
applebot
applebot-extended
brightbot 1.0
bytespider
ccbot
chatgpt-user
claude-web
claudebot
cohere-ai
cohere-training-data-crawler
crawlspace
diffbot
duckassistbot
facebookbot
friendlycrawler
google-extended
googleother
googleother-image
googleother-video
gptbot
iaskspider/2.0
icc-crawler
imagesiftbot
img2dataset
isscyberriskcrawler
kangaroo bot
meta-externalagent
meta-externalfetcher
oai-searchbot
omgili
omgilibot
pangubot
perplexitybot
perplexity‑user
petalbot
scrapy
semrushbot-ocob
semrushbot-swa
sidetrade indexer bot
timpibot
velenpublicwebcrawler
webzio-extended
youbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.michaelchell.co.uk/pub/sitemap.xml

Comments

  • Allow all user-agents
  • Block access to the following sensitive directories
  • Block admin pages and other internal URLs
  • Block URLs for sorting, filtering, and search result pages
  • Block common query strings
  • Allow indexing of important pages
  • Block AhrefsBot