thelist.com
robots.txt
Robots Exclusion Standard data for thelist.com
Resource Scan
Scan Details
| Site Domain | thelist.com |
| Base Domain | thelist.com |
| Scan Status | Ok |
| Last Scan | 2024-05-25T21:18:07+00:00 |
| Next Scan | 2024-06-01T21:18:07+00:00 |
Last Scan
| Scanned | 2024-05-25T21:18:07+00:00 |
| URL | https://thelist.com/robots.txt |
| Domain IPs | 108.157.254.10, 108.157.254.14, 108.157.254.20, 108.157.254.90 |
| Response IP | 108.157.254.20 |
| Found | Yes |
| Hash | 9c6f6a00aae52205fda8e48685392006ee0a36438cfb410f9f277677da483619 |
| SimHash | 7a045848ab93 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /wp-admin/ |
| Disallow | /wp-includes/ |
| Disallow | /*?*ajax= |
| Disallow | /*/s/* |
| Disallow | /*/sl/* |
| Disallow | /search/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.thelist.com/sitemap_index.xml |
| sitemap | https://www.thelist.com/stories/sitemap-index.xml |
| sitemap | https://www.thelist.com/?getfeed=google |