curi.us
robots.txt

Robots Exclusion Standard data for curi.us

Resource Scan

Scan Details

Site Domain curi.us
Base Domain curi.us
Scan Status Ok
Last Scan2025-08-27T12:16:42+00:00
Next Scan 2025-09-26T12:16:42+00:00

Last Scan

Scanned2025-08-27T12:16:42+00:00
URL https://curi.us/robots.txt
Domain IPs 104.21.5.253, 172.67.134.24, 2606:4700:3030::6815:5fd, 2606:4700:3032::ac43:8618
Response IP 104.21.5.253
Found Yes
Hash 6126eda5c221a6c37e39cc782493874698cd3145849bcd7eb9e331c829375670
SimHash 6b1c9862fbb3

Groups

*

Rule Path
Disallow
Allow /

ahrefsbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 90

mj12bot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 90

semrushbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 90

Other Records

Field Value
sitemap https://curi.us/sitemap.txt

Comments

  • Crawl-Delay: 1