deviantart.com
robots.txt
Robots Exclusion Standard data for deviantart.com
Resource Scan
Scan Details
| Site Domain | deviantart.com |
| Base Domain | deviantart.com |
| Scan Status | Ok |
| Last Scan | 2024-11-12T08:03:18+00:00 |
| Next Scan | 2024-11-19T08:03:18+00:00 |
Last Scan
| Scanned | 2024-11-12T08:03:18+00:00 |
| URL | https://deviantart.com/robots.txt |
| Redirect | https://www.deviantart.com/robots.txt |
| Redirect Domain | www.deviantart.com |
| Redirect Base | deviantart.com |
| Domain IPs | 13.33.88.103, 13.33.88.12, 13.33.88.35, 13.33.88.43 |
| Redirect IPs | 13.33.88.103, 13.33.88.12, 13.33.88.35, 13.33.88.43 |
| Response IP | 13.33.88.103 |
| Found | Yes |
| Hash | a5c90c8708fea40e07996cc8abd476ef7db4028e210df4ba329c88b28bc5b93f |
| SimHash | 7d08d87288a7 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /users/*? |
| Disallow | /join/*? |
| Disallow | /morelikethis/ |
| Disallow | /download/ |
| Disallow | /checkout/ |
| Disallow | /global/ |
| Disallow | /api/ |
| Disallow | /critiques/ |
| Disallow | /prints/*by%3A* |
| Disallow | /_stash/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.deviantart.com/sitemap-index.xml.gz |