buzzfeed.com
robots.txt
Robots Exclusion Standard data for buzzfeed.com
Resource Scan
Scan Details
| Site Domain | buzzfeed.com |
| Base Domain | buzzfeed.com |
| Scan Status | Ok |
| Last Scan | 2024-11-09T13:05:05+00:00 |
| Next Scan | 2024-11-16T13:05:05+00:00 |
Last Scan
| Scanned | 2024-11-09T13:05:05+00:00 |
| URL | https://buzzfeed.com/robots.txt |
| Redirect | https://www.buzzfeed.com/robots.txt |
| Redirect Domain | www.buzzfeed.com |
| Redirect Base | buzzfeed.com |
| Domain IPs | 151.101.130.114, 151.101.194.114, 151.101.2.114, 151.101.66.114 |
| Redirect IPs | 151.101.130.114, 151.101.194.114, 151.101.2.114, 151.101.66.114 |
| Response IP | 199.232.46.114 |
| Found | Yes |
| Hash | 36b0dd7e2efa604a68a37097e01ac63fe17566c8ac3737799a7c1debdfe46e25 |
| SimHash | 313a5d518c81 |
Groups
msnbot
| Rule | Path |
|---|---|
| Disallow | /*.xml$ |
| Disallow | /buzz/*.xml$ |
| Disallow | /category/*.xml$ |
| Disallow | /mobile/ |
| Disallow | *?s=mobile |
| Disallow | *?s=bpage-next |
| Disallow | *?s=lightbox |
| Disallow | /fabordrab/ |
| Disallow | /bfmp/ |
| Disallow | /buzzfeed/ |
| Disallow | /contest |
| Disallow | /contests |
| Disallow | /plugin/ |
| Disallow | /embed/ |
| Disallow | /_comments/ |
| Disallow | /bookmarks/ |
| Disallow | /drafts/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 120 |
*
| Rule | Path |
|---|---|
| Disallow | /buzz/*.xml$ |
| Disallow | /category/*.xml$ |
| Disallow | /mobile/ |
| Disallow | *?s=bpage-next |
| Disallow | *?s=lightbox |
| Disallow | *?s=feedpager |
| Disallow | /fabordrab/ |
| Disallow | /bfmp/ |
| Disallow | /buzzfeed/ |
| Disallow | /contest |
| Disallow | /contests |
| Disallow | /_ga/ |
| Disallow | /static/ |
| Disallow | /dashboard/ |
| Disallow | /plugin/ |
| Disallow | /api/ |
| Disallow | /buzzfeed/api/ |
| Disallow | /embed/ |
| Disallow | /_comments/ |
| Disallow | /bookmarks/ |
| Disallow | /drafts/ |
| Disallow | /bpage-preview/ |
| Disallow | /cms_preview_ui/ |
| Disallow | /cms_preview_ui_static/ |
| Disallow | /search/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.buzzfeed.com/sitemap/asis.xml |
| sitemap | https://www.buzzfeed.com/sitemap/buzzfeed.xml |
| sitemap | https://www.buzzfeed.com/sitemap/tasty.xml |
| sitemap | https://www.buzzfeed.com/sitemap/video.xml |
| sitemap | https://www.buzzfeed.com/sitemap/shopping.xml |
| sitemap | https://www.buzzfeed.com/sitemap/topics.xml |
| sitemap | https://www.buzzfeed.com/sitemap/travel.xml |
| sitemap | https://www.buzzfeed.com/sitemap/news.xml |
| sitemap | https://www.buzzfeed.com/sitemap/buzzfeed-community.xml |
| sitemap | https://www.buzzfeed.com/sitemap/editions.xml |
| sitemap | https://www.buzzfeed.com/webstories/sitemap.xml |