censusreporter.org
robots.txt

Robots Exclusion Standard data for censusreporter.org

Resource Scan

Scan Details

Site Domain censusreporter.org
Base Domain censusreporter.org
Scan Status Ok
Last Scan2024-04-29T20:20:51+00:00
Next Scan 2024-05-29T20:20:51+00:00

Last Scan

Scanned2024-04-29T20:20:51+00:00
URL https://censusreporter.org/robots.txt
Domain IPs 54.230.21.122, 54.230.21.36, 54.230.21.72, 54.230.21.9
Response IP 18.165.171.55
Found Yes
Hash 60f937f063f5bf5f0b90e1783e026886cfe99a3fbd3445146d701d852b45202b
SimHash 4d0c485047b7

Groups

*

Rule Path
Allow /
Disallow /data/

Other Records

Field Value
crawl-delay 3

Other Records

Field Value
sitemap https://censusreporter.org/static/sitemap/sitemap.xml