thinker.org
robots.txt

Robots Exclusion Standard data for thinker.org

Resource Scan

Scan Details

Site Domain thinker.org
Base Domain thinker.org
Scan Status Ok
Last Scan2025-12-05T09:37:31+00:00
Next Scan 2026-01-04T09:37:31+00:00

Last Scan

Scanned2025-12-05T09:37:31+00:00
URL https://thinker.org/robots.txt
Redirect https://www.famsf.org/robots.txt
Redirect Domain www.famsf.org
Redirect Base famsf.org
Domain IPs 151.101.66.216
Redirect IPs 151.101.130.216, 151.101.194.216, 151.101.2.216, 151.101.66.216
Response IP 151.101.194.216
Found Yes
Hash 9ee2f1c63f836c13f4d2cb567f32fd4ceccd8b340d688b0a3d049884c7ef7ae3
SimHash 624cc800e300

Groups

*

Rule Path
Disallow /art/*
Disallow /about/rent-the-museums/rental-event-inquiry-form
Disallow /about/rent-the-museums/confirmation

amazonbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

viberbot

Rule Path
Disallow /

yak

Rule Path
Disallow /

yandex

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10