Log In Sign Up

cosmostation.io
robots.txt

Robots Exclusion Standard data for cosmostation.io

Resource Scan

Scan Details

Site Domain cosmostation.io
Base Domain cosmostation.io
Scan Status Ok
Last Scan2024-09-22T01:42:25+00:00
Next Scan 2024-10-22T01:42:25+00:00

Last Scan

Scanned2024-09-22T01:42:25+00:00
URL https://cosmostation.io/robots.txt
Domain IPs 108.156.133.108, 108.156.133.116, 108.156.133.13, 108.156.133.82
Response IP 108.156.133.82
Found Yes
Hash 76f874033533d8c86ab8eefc1db1f34cca3da410dd5886fbc554005e073f5dbe
SimHash ac101510c9e1

Groups

*

Rule Path
Disallow /

googlebot

Rule Path
Allow /

yeti

Rule Path
Allow /

slackbot 1.0

Rule Path
Allow /

slackbot-linkexpanding 1.0

Rule Path
Allow /

twitterbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /
Back to top

Comments

  • Disallow everything.
  • Allow googlebot
  • Allow naverbot
  • Certain social media sites are whitelisted to allow crawlers to access page markup when links to /images are shared.
Back to top