spreeder.com
robots.txt

Robots Exclusion Standard data for spreeder.com

Resource Scan

Scan Details

Site Domain spreeder.com
Base Domain spreeder.com
Scan Status Ok
Last Scan2026-01-29T15:58:39+00:00
Next Scan 2026-02-28T15:58:39+00:00

Last Scan

Scanned2026-01-29T15:58:39+00:00
URL https://spreeder.com/robots.txt
Domain IPs 104.26.14.145, 104.26.15.145, 172.67.71.186, 2606:4700:20::681a:e91, 2606:4700:20::681a:f91, 2606:4700:20::ac43:47ba
Response IP 172.67.71.186
Found Yes
Hash f29d706b90440b5068aeac679810712e2d6a427fe02e5f282fc361af46b297e0
SimHash e58b58464fd3

Groups

*

Rule Path
Disallow
Disallow /cgi-bin/

Other Records

Field Value
crawl-delay 20

Other Records

Field Value
sitemap https://www.spreeder.com/sitemap.xml

Warnings

  • 1 invalid line.