cnet.com
robots.txt
Robots Exclusion Standard data for cnet.com
Resource Scan
Scan Details
| Site Domain | cnet.com |
| Base Domain | cnet.com |
| Scan Status | Ok |
| Last Scan | 2024-04-26T12:04:21+00:00 |
| Next Scan | 2024-05-03T12:04:21+00:00 |
Last Scan
| Scanned | 2024-04-26T12:04:21+00:00 |
| URL | https://cnet.com/robots.txt |
| Redirect | https://www.cnet.com/robots.txt |
| Redirect Domain | www.cnet.com |
| Redirect Base | cnet.com |
| Domain IPs | 34.149.196.126 |
| Redirect IPs | 199.232.194.154, 199.232.198.154, 2a04:4e42:4c::666, 2a04:4e42:4d::666 |
| Response IP | 146.75.42.154 |
| Found | Yes |
| Hash | 11176183b18c21924a43a4158b23e62533e46a55d682c0cc275dfe32791b3482 |
| SimHash | c807f25ce333 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | */*max-price%3D* |
| Disallow | */*min-price%3D* |
| Disallow | */Xhr* |
| Disallow | */account-information/* |
| Disallow | */ad/ad-cookie/* |
| Disallow | */ajax/ |
| Disallow | */alert/* |
| Disallow | */cnet_global_app_redirect.xml |
| Disallow | */component/* |
| Disallow | */edit/ |
| Disallow | */edit/change-email/* |
| Disallow | */edit/change-password/* |
| Disallow | */email/editor/* |
| Disallow | */email/member/* |
| Disallow | */email/newsletter/* |
| Disallow | */error/* |
| Disallow | */ext/make/ingest/* |
| Disallow | */feature/* |
| Disallow | */finder/*/*/?*filter=*_*_*_*_* |
| Disallow | */finder/*/?*filter=*_*_*_*_* |
| Disallow | */forums/* |
| Disallow | */g00/* |
| Disallow | */informacion-de-mi-cuenta/* |
| Disallow | */members/* |
| Disallow | */products/*?*filter=* |
| Disallow | */roadshow/finder/* |
| Disallow | */search/* |
| Disallow | */user/* |
| Disallow | */videos/manifest/* |
| Disallow | */videos/secure/* |
| Disallow | */videos/selector/* |
| Disallow | */wrap-function* |
| Disallow | */wrap-xhr* |
| Disallow | */xhr* |
| Disallow | *?sort=popularity* |
| Disallow | *?sort=price* |
| Disallow | *?sort=user-rating* |
| Disallow | *hub-preview-version%3D* |
| Disallow | /.well-known/* |
| Disallow | /api/editionurlgenerator/ |
| Disallow | /apple-news/* |
| Disallow | /au/* |
| Disallow | /es/* |
| Disallow | /html/story/* |
| Disallow | /index.php/* |
| Disallow | /products/* |
| Disallow | /uk/* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.cnet.com/sitemaps/article/index.xml |
| sitemap | https://www.cnet.com/sitemaps/best/index.xml |
| sitemap | https://www.cnet.com/sitemaps/gallery/index.xml |
| sitemap | https://www.cnet.com/sitemaps/how-to/index.xml |
| sitemap | https://www.cnet.com/sitemaps/news.xml |
| sitemap | https://www.cnet.com/sitemaps/review/index.xml |
| sitemap | https://www.cnet.com/sitemaps/static.xml |
| sitemap | https://www.cnet.com/sitemaps/updating_story/index.xml |
| sitemap | https://www.cnet.com/sitemaps/video/index.xml |
| sitemap | https://www.cnet.com/personal-finance/sitemaps/index.xml |
Comments