Log In Sign Up

cantookstation.eu
robots.txt

Robots Exclusion Standard data for cantookstation.eu

Resource Scan

Scan Details

Site Domain cantookstation.eu
Base Domain cantookstation.eu
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-01-04T21:12:10+00:00
Next Scan 2026-03-05T21:12:10+00:00

Last Successful Scan

Scanned2025-10-13T23:39:07+00:00
URL https://cantookstation.eu/robots.txt
Domain IPs 34.107.57.161
Response IP 34.107.57.161
Found Yes
Hash 3a42b48e640402314bee9e9c81757ad9739896a0970ad2bb6e4000cc897b78b7
SimHash 3875d351c5d5

Groups

applebot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

googlebot

Rule Path
Allow /

*

Rule Path
Disallow /
Back to top

Comments

  • Applebot is the web crawler for Apple.
  • https://www.apple.com/go/applebot
  • Bing search engine robot.
  • https://www.bing.com/bingbot.htm
  • Petal search engine Robot (HUAWEI).
  • https://aspiegel.com/petalbot
  • SEMrushBot is the search bot software used by SEMrush.
  • https://www.semrush.com/bot.html
  • Googlebot is the generic name for Google's web crawler.
  • https://support.google.com/webmasters/answer/182072
  • All other robots are disallowed from crawling our servers.
Back to top