buzzfeednews.com
robots.txt
Robots Exclusion Standard data for buzzfeednews.com
Resource Scan
Scan Details
| Site Domain | buzzfeednews.com |
| Base Domain | buzzfeednews.com |
| Scan Status | Ok |
| Last Scan | 2024-10-30T12:59:15+00:00 |
| Next Scan | 2024-11-06T12:59:15+00:00 |
Last Scan
| Scanned | 2024-10-30T12:59:15+00:00 |
| URL | https://buzzfeednews.com/robots.txt |
| Redirect | https://www.buzzfeednews.com/robots.txt |
| Redirect Domain | www.buzzfeednews.com |
| Redirect Base | buzzfeednews.com |
| Domain IPs | 151.101.130.114, 151.101.194.114, 151.101.2.114, 151.101.66.114 |
| Redirect IPs | 151.101.130.114, 151.101.194.114, 151.101.2.114, 151.101.66.114 |
| Response IP | 199.232.46.114 |
| Found | Yes |
| Hash | 4098829626c94e4b7f6b5d99ec6d9e0f7be8f8bac54209974f8d330f4ac0c998 |
| SimHash | 7b3acd708ea1 |
Groups
msnbot
| Rule | Path |
|---|---|
| Disallow | /*.xml$ |
| Disallow | /category/*.xml$ |
| Disallow | /mobile/ |
| Disallow | *?s=mobile |
| Disallow | *?s=lightbox |
| Disallow | /buzzfeed/ |
| Disallow | /contest |
| Disallow | /contests |
| Disallow | /plugin/ |
| Disallow | /embed/ |
| Disallow | /_comments/ |
| Disallow | /bookmarks/ |
| Disallow | /drafts/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 120 |
*
| Rule | Path |
|---|---|
| Disallow | /buzz/*.xml$ |
| Disallow | /category/*.xml$ |
| Disallow | /mobile/ |
| Disallow | *?s=lightbox |
| Disallow | *?s=feedpager |
| Disallow | /buzzfeed/ |
| Disallow | /contest |
| Disallow | /contests |
| Disallow | /_ga/ |
| Disallow | /static/ |
| Disallow | /dashboard/ |
| Disallow | /plugin/ |
| Disallow | /api/ |
| Disallow | /buzzfeed/api/ |
| Disallow | /embed/ |
| Disallow | /_comments/ |
| Disallow | /bookmarks/ |
| Disallow | /drafts/ |
| Disallow | /bpage-preview/ |
| Disallow | /feed/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.buzzfeednews.com/sitemap/news.xml |
| sitemap | https://www.buzzfeednews.com/sitemap/recent.xml |
| sitemap | https://www.buzzfeednews.com/sitemap/sections.xml |
| sitemap | https://www.buzzfeednews.com/sitemap/topics.xml |