Log In Sign Up

clevercookie.uk
robots.txt

Robots Exclusion Standard data for clevercookie.uk

Resource Scan

Scan Details

Site Domain clevercookie.uk
Base Domain clevercookie.uk
Scan Status Ok
Last Scan2026-03-18T20:10:17+00:00
Next Scan 2026-04-17T20:10:17+00:00

Last Scan

Scanned2026-03-18T20:10:17+00:00
URL https://clevercookie.uk/robots.txt
Redirect https://www.clevercookie.uk/robots.txt
Redirect Domain www.clevercookie.uk
Redirect Base clevercookie.uk
Domain IPs 104.26.14.71, 104.26.15.71, 172.67.71.232, 2606:4700:20::681a:e47, 2606:4700:20::681a:f47, 2606:4700:20::ac43:47e8
Redirect IPs 104.26.14.71, 104.26.15.71, 172.67.71.232, 2606:4700:20::681a:e47, 2606:4700:20::681a:f47, 2606:4700:20::ac43:47e8
Response IP 172.67.71.232
Found Yes
Hash ebffedafe16cecbe440b630eb1dee16e55c1e0eb7434f0d01892424ed20090fc
SimHash 7d358b14c8d3

Groups

gptbot
chatgpt-user
oai-searchbot
claudebot
claude-web
perplexitybot
google-extended
applebot-extended

Rule Path
Allow /
Disallow /api/
Disallow /cart
Disallow /account
Disallow /search
Disallow /review
Disallow /corporate/checkout
Disallow /corporate/confirmation

*

Rule Path
Allow /
Disallow /api/
Disallow /cart
Disallow /account
Disallow /search
Disallow /review
Disallow /corporate/checkout
Disallow /corporate/confirmation
Back to top

Other Records

Field Value
sitemap https://www.clevercookie.uk/sitemap.xml
Back to top

Comments

  • AI Crawlers - explicitly welcomed, same rules as all other crawlers
  • LLM-friendly content
Back to top

Warnings

  • `llms-txt` is not a known field.
Back to top