fanfiction.net
robots.txt
Robots Exclusion Standard data for fanfiction.net
Resource Scan
Scan Details
| Site Domain | fanfiction.net |
| Base Domain | fanfiction.net |
| Scan Status | Ok |
| Last Scan | 2024-09-14T14:28:36+00:00 |
| Next Scan | 2024-09-21T14:28:36+00:00 |
Last Scan
| Scanned | 2024-09-14T14:28:36+00:00 |
| URL | https://fanfiction.net/robots.txt |
| Redirect | https://www.fanfiction.net/robots.txt |
| Redirect Domain | www.fanfiction.net |
| Redirect Base | fanfiction.net |
| Domain IPs | 143.244.211.110, 174.138.110.4 |
| Redirect IPs | 104.18.18.242, 104.18.19.242, 2606:4700::6812:12f2, 2606:4700::6812:13f2 |
| Response IP | 104.18.18.242 |
| Found | Yes |
| Hash | 31802f3f86b774219b6de732ed10d6d4defb65fab38fe16f6aa56c97cea7652e |
| SimHash | 0524cc504353 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /secure/ |
| Disallow | /rs/ |
| Disallow | /ru/ |
| Disallow | /eye/ |
| Disallow | /m/ |
| Disallow | /*.php |
| Allow | / |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |