classaction.org
robots.txt

Robots Exclusion Standard data for classaction.org

Resource Scan

Scan Details

Site Domain classaction.org
Base Domain classaction.org
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-21T13:51:07+00:00
Next Scan 2024-12-20T13:51:07+00:00

Last Successful Scan

Scanned2023-05-28T13:56:24+00:00
URL https://classaction.org/robots.txt
Redirect https://www.classaction.org/robots.txt
Redirect Domain www.classaction.org
Redirect Base classaction.org
Domain IPs 172.66.40.209, 172.66.43.47, 2606:4700:3108::ac42:28d1, 2606:4700:3108::ac42:2b2f
Redirect IPs 172.66.40.209, 172.66.43.47, 2606:4700:3108::ac42:28d1, 2606:4700:3108::ac42:2b2f
Response IP 172.66.40.209
Found Yes
Hash be0d069a4fc5f83c6e8914aa22af1b384cf73b75461c9a6bfc351c881997e213
SimHash 0f44d9f3f40a

Groups

*

Rule Path
Allow /

*

Rule Path
Disallow /*.php

*

Rule Path
Disallow /api/

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

spbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider-video

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

yandex

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

typhoeus

Rule Path
Disallow /

node/simplecrawler*

Rule Path
Disallow /

xenu

Rule Path
Disallow /

ltx71

Rule Path
Disallow /