skyscript.co.uk
robots.txt

Robots Exclusion Standard data for skyscript.co.uk

Resource Scan

Scan Details

Site Domain skyscript.co.uk
Base Domain skyscript.co.uk
Scan Status Ok
Last Scan2025-04-26T09:18:33+00:00
Next Scan 2025-05-26T09:18:33+00:00

Last Scan

Scanned2025-04-26T09:18:33+00:00
URL https://skyscript.co.uk/robots.txt
Domain IPs 104.21.63.105, 172.67.170.143, 2606:4700:3032::6815:3f69, 2606:4700:3036::ac43:aa8f
Response IP 172.67.170.143
Found Yes
Hash ceb3f38151e6e536ed176255bfc98eedfe85f2741b12868cb412b0f76b5a70c5
SimHash 62b0d151c4c2

Groups

*

Rule Path
Disallow /im/
Disallow /ads/
Disallow /ban/
Disallow /cgi-bin/
Disallow /extras/
Disallow /forms/
Disallow /images/
Disallow /img/
Disallow /pdf/
Disallow /phpBB2/
Disallow /hotlinks.html
Disallow /topbanner.html

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

perplexity-archive-bot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

Comments

  • list folders robots are not allowed to index
  • list specific files robots are not allowed to index
  • Crawl-delay directive (some bots honor this)