contagiousmedia.com
robots.txt

Robots Exclusion Standard data for contagiousmedia.com

Resource Scan

Scan Details

Site Domain contagiousmedia.com
Base Domain contagiousmedia.com
Scan Status Ok
Last Scan2024-10-28T17:26:49+00:00
Next Scan 2024-11-04T17:26:49+00:00

Last Scan

Scanned2024-10-28T17:26:49+00:00
URL https://contagiousmedia.com/robots.txt
Redirect https://www.buzzfeed.com/robots.txt
Redirect Domain www.buzzfeed.com
Redirect Base buzzfeed.com
Domain IPs 13.35.210.112, 13.35.210.2, 13.35.210.23, 13.35.210.74
Redirect IPs 151.101.130.114, 151.101.194.114, 151.101.2.114, 151.101.66.114
Response IP 199.232.46.114
Found Yes
Hash 36b0dd7e2efa604a68a37097e01ac63fe17566c8ac3737799a7c1debdfe46e25
SimHash 313a5d518c81

Groups

msnbot

Rule Path
Disallow /*.xml$
Disallow /buzz/*.xml$
Disallow /category/*.xml$
Disallow /mobile/
Disallow *?s=mobile
Disallow *?s=bpage-next
Disallow *?s=lightbox
Disallow /fabordrab/
Disallow /bfmp/
Disallow /buzzfeed/
Disallow /contest
Disallow /contests
Disallow /plugin/
Disallow /embed/
Disallow /_comments/
Disallow /bookmarks/
Disallow /drafts/

Other Records

Field Value
crawl-delay 120

*

Rule Path
Disallow /buzz/*.xml$
Disallow /category/*.xml$
Disallow /mobile/
Disallow *?s=bpage-next
Disallow *?s=lightbox
Disallow *?s=feedpager
Disallow /fabordrab/
Disallow /bfmp/
Disallow /buzzfeed/
Disallow /contest
Disallow /contests
Disallow /_ga/
Disallow /static/
Disallow /dashboard/
Disallow /plugin/
Disallow /api/
Disallow /buzzfeed/api/
Disallow /embed/
Disallow /_comments/
Disallow /bookmarks/
Disallow /drafts/
Disallow /bpage-preview/
Disallow /cms_preview_ui/
Disallow /cms_preview_ui_static/
Disallow /search/

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 4

yacybot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.buzzfeed.com/sitemap/asis.xml
sitemap https://www.buzzfeed.com/sitemap/buzzfeed.xml
sitemap https://www.buzzfeed.com/sitemap/tasty.xml
sitemap https://www.buzzfeed.com/sitemap/video.xml
sitemap https://www.buzzfeed.com/sitemap/shopping.xml
sitemap https://www.buzzfeed.com/sitemap/topics.xml
sitemap https://www.buzzfeed.com/sitemap/travel.xml
sitemap https://www.buzzfeed.com/sitemap/news.xml
sitemap https://www.buzzfeed.com/sitemap/buzzfeed-community.xml
sitemap https://www.buzzfeed.com/sitemap/editions.xml
sitemap https://www.buzzfeed.com/webstories/sitemap.xml