cals.ncsu.edu
robots.txt

Robots Exclusion Standard data for cals.ncsu.edu

Resource Scan

Scan Details

Site Domain cals.ncsu.edu
Base Domain ncsu.edu
Scan Status Ok
Last Scan2025-03-11T16:08:13+00:00
Next Scan 2025-04-10T16:08:13+00:00

Last Scan

Scanned2025-03-11T16:08:13+00:00
URL https://cals.ncsu.edu/robots.txt
Domain IPs 152.7.102.42
Response IP 152.7.102.42
Found Yes
Hash 6860095167f635c3d8781030363727b77268e4851c7229ac584c837c6b0706a9
SimHash c820c0534372

Groups

*

Rule Path
Disallow /vetpac/my-vetpac/*

Other Records

Field Value
crawl-delay 5

semanticscholarbot

Rule Path
Disallow /*.pdf

Other Records

Field Value
crawl-delay 10

paracrawl

Rule Path
Disallow /

googlebot

Rule Path
Allow .js
Allow .css
Allow .png
Allow .jpg