classicthesaurus.com
robots.txt

Robots Exclusion Standard data for classicthesaurus.com

Resource Scan

Scan Details

Site Domain classicthesaurus.com
Base Domain classicthesaurus.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-07T15:23:55+00:00
Next Scan 2026-01-05T15:23:55+00:00

Last Successful Scan

Scanned2025-06-10T14:16:15+00:00
URL https://classicthesaurus.com/robots.txt
Redirect https://www.classicthesaurus.com/robots.txt
Redirect Domain www.classicthesaurus.com
Redirect Base classicthesaurus.com
Domain IPs 104.21.14.213, 172.67.160.149, 2606:4700:3030::ac43:a095, 2606:4700:3035::6815:ed5
Redirect IPs 104.21.14.213, 172.67.160.149, 2606:4700:3030::ac43:a095, 2606:4700:3035::6815:ed5
Response IP 172.67.160.149
Found Yes
Hash 88888e0bc180184ab8853949542b859c49aaae93e7c5b6f9d6ccf3baaeadcef6
SimHash 6a4ad160d335

Groups

*

Rule Path
Disallow /_search
Disallow /_add

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

grapeshot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1