dev.thecrimson.com
robots.txt

Robots Exclusion Standard data for dev.thecrimson.com

Resource Scan

Scan Details

Site Domain dev.thecrimson.com
Base Domain thecrimson.com
Scan Status Ok
Last Scan2024-05-07T09:47:59+00:00
Next Scan 2024-06-06T09:47:59+00:00

Last Scan

Scanned2024-05-07T09:47:59+00:00
URL https://dev.thecrimson.com/robots.txt
Domain IPs 18.164.174.11, 18.164.174.126, 18.164.174.17, 18.164.174.64
Response IP 18.165.171.82
Found Yes
Hash 86d88d40bc2c427d29729f0db339f3a974e7f57b3adf7398134ebdda34a27869
SimHash ec2a5335c393

Groups

hul-wax

Rule Path
Disallow

slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5

yahoo! slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5