api.thecrimson.com
robots.txt

Robots Exclusion Standard data for api.thecrimson.com

Resource Scan

Scan Details

Site Domain api.thecrimson.com
Base Domain thecrimson.com
Scan Status Ok
Last Scan2024-09-22T08:00:25+00:00
Next Scan 2024-10-22T08:00:25+00:00

Last Scan

Scanned2024-09-22T08:00:25+00:00
URL https://api.thecrimson.com/robots.txt
Domain IPs 3.221.251.152, 54.156.179.10
Response IP 54.156.179.10
Found Yes
Hash 86d88d40bc2c427d29729f0db339f3a974e7f57b3adf7398134ebdda34a27869
SimHash ec2a5335c393

Groups

hul-wax

Rule Path
Disallow

slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5

yahoo! slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5