pentastars.com
robots.txt

Robots Exclusion Standard data for pentastars.com

Resource Scan

Scan Details

Site Domain pentastars.com
Base Domain pentastars.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-11-12T16:15:18+00:00
Next Scan 2024-11-19T16:15:18+00:00

Last Successful Scan

Scanned2024-04-22T04:37:22+00:00
URL https://pentastars.com/robots.txt
Redirect https://www.pentastars.com/robots.txt
Redirect Domain www.pentastars.com
Redirect Base pentastars.com
Domain IPs 151.101.1.91, 151.101.129.91, 151.101.193.91, 151.101.65.91
Redirect IPs 151.101.1.91, 151.101.129.91, 151.101.193.91, 151.101.65.91
Response IP 199.232.45.91
Found Yes
Hash 51f1db8564e892302f08e19f8010991ac1c0560c599173ce903960018ac343e5
SimHash 6f41d414c315

Groups

slurp

Rule Path
Disallow /admin/

Other Records

Field Value
crawl-delay 3

msn

Rule Path
Disallow /admin/

Other Records

Field Value
crawl-delay 5

msnbot

Rule Path
Disallow /admin/

Other Records

Field Value
crawl-delay 5

yandex

Rule Path
Disallow /forums/min/

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /cgi-bin/