astrologify.com
robots.txt
Robots Exclusion Standard data for astrologify.com
Resource Scan
Scan Details
| Site Domain | astrologify.com |
| Base Domain | astrologify.com |
| Scan Status | Ok |
| Last Scan | 2024-10-04T14:55:54+00:00 |
| Next Scan | 2024-10-11T14:55:54+00:00 |
Last Scan
| Scanned | 2024-10-04T14:55:54+00:00 |
| URL | https://astrologify.com/robots.txt |
| Domain IPs | 104.21.82.165, 172.67.159.209, 2606:4700:3036::6815:52a5, 2606:4700:3037::ac43:9fd1 |
| Response IP | 104.21.82.165 |
| Found | Yes |
| Hash | 0990d7f37d75f305505aace6ea4abba838bb0aebae131882ccf34a0620e35c3b |
| SimHash | 6f085e55ca13 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /admin/ |
| Disallow | /crm/ |
| Disallow | /go/ |
| Disallow | /member/ |
| Disallow | /xmlrpc.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 600 |
Other Records
| Field | Value |
|---|---|
| sitemap | http://astrologify.com/stmp/s.xml |
Warnings
- `noindex` is not a known field.