Log In Sign Up

beyondfossilfuels.org
robots.txt

Robots Exclusion Standard data for beyondfossilfuels.org

Resource Scan

Scan Details

Site Domain beyondfossilfuels.org
Base Domain beyondfossilfuels.org
Scan Status Ok
Last Scan2025-12-06T09:27:31+00:00
Next Scan 2026-01-05T09:27:31+00:00

Last Scan

Scanned2025-12-06T09:27:31+00:00
URL https://beyondfossilfuels.org/robots.txt
Domain IPs 104.26.4.16, 104.26.5.16, 172.67.72.15, 2606:4700:20::681a:410, 2606:4700:20::681a:510, 2606:4700:20::ac43:480f
Response IP 104.26.4.16
Found Yes
Hash 9e82c8b952dbdb11ccd9d313760ec9dfee234133fe39e0d5113d694bef20d04e
SimHash 401cd240e511

Groups

googlebot

Rule Path
Disallow

bingbot

Rule Path
Disallow

slurp

Rule Path
Disallow

duckduckbot

Rule Path
Disallow

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

adsbot-google

Rule Path
Disallow /

sogou

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

applebot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

*

Rule Path
Disallow /
Back to top

Comments

  • Allow major search engines
  • Block all AI crawlers
  • Block all others by default
Back to top