goatbots.com
robots.txt
Robots Exclusion Standard data for goatbots.com
Resource Scan
Scan Details
| Site Domain | goatbots.com |
| Base Domain | goatbots.com |
| Scan Status | Ok |
| Last Scan | 2025-10-31T21:10:28+00:00 |
| Next Scan | 2025-11-30T21:10:28+00:00 |
Last Scan
| Scanned | 2025-10-31T21:10:28+00:00 |
| URL | https://goatbots.com/robots.txt |
| Redirect | https://www.goatbots.com/robots.txt |
| Redirect Domain | www.goatbots.com |
| Redirect Base | goatbots.com |
| Domain IPs | 172.66.40.86, 172.66.43.170, 2606:4700:3108::ac42:2856, 2606:4700:3108::ac42:2baa |
| Redirect IPs | 172.66.40.86, 172.66.43.170, 2606:4700:3108::ac42:2856, 2606:4700:3108::ac42:2baa |
| Response IP | 172.66.43.170 |
| Found | Yes |
| Hash | e0fb7a456a4d790a8a5e431a40b79c16ab143b7b2c7bf0b36d6ef8d8014ee62f |
| SimHash | 2513c8c0e791 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /ajax/ |
| Disallow | /download/ |
| Disallow | /cdn-cgi/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 3 |