Log In Sign Up

food.com
robots.txt

Robots Exclusion Standard data for food.com

Resource Scan

Scan Details

Site Domain food.com
Base Domain food.com
Scan Status Ok
Last Scan2024-09-22T18:33:52+00:00
Next Scan 2024-09-29T18:33:52+00:00

Last Scan

Scanned2024-09-22T18:33:52+00:00
URL https://www.food.com/robots.txt
Domain IPs 23.210.106.27
Response IP 23.77.31.218
Found Yes
Hash 9475680d065be74787ff6f2d0903c673eb90fe70e4eeec7869205622dc5badcb
SimHash 403079d6c531

Groups

*

Rule Path
Disallow /recipeprint.do*
Disallow /rssapi.do*
Disallow /bb/
Disallow /null/
Disallow /members/subm/
Disallow /mail/compose*
Disallow /cp_asset_stats.do
Disallow /dpn.do*
Disallow /mobile/api/
Disallow /ingredient-finder/
Disallow /cookbook-finder/
Disallow /menu-finder/
Disallow /recipe-finder/
Disallow /photo-finder/
Disallow /member-finder/
Disallow /search/
Disallow /books/
Disallow /community/
Disallow *.zsp*
Disallow *.php*
Disallow /~asset*

mediapartners-google

Rule Path
Disallow

googlebot-news

Rule Path
Allow /news
Allow /article
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.food.com/sitemap.xml
sitemap https://www.food.com/news/news-sitemap.xml
Back to top

Comments

  • Directives for all crawlers
  • Disallow: /about/tour/
  • Disallow: /admin/
  • Disallow: /alt/
  • Disallow: /mail/
  • Disallow: /members/photos.php
  • Disallow: /recipe/photo.php
  • Disallow: /void/
  • Disallow: /services/
  • Exclude page construction elements (commented out some due to new google guidelines)
  • Disallow: /pushdownAd.do
  • Disallow: /global/header.jsp
  • Disallow: /global/pickle-header.jsp
  • Disallow: /footer.jsp
  • Disallow: /headerbreadcrumbs.do
  • Disallow: /getrzuserstats.do
  • Disallow: /seasonal-header.do
  • Exclude authenticated pages
  • Exclude search results pages
  • Block invalid Omniture injected URLs within javascript
  • Disallow: /*_1
  • Disallow: /*_2
  • Disallow: /*_3
  • Disallow: /*_4
  • Disallow: /*_5
  • Disallow: /*_6
  • Disallow: /*_7
  • Disallow: /*_8
  • Block .php & .zsp files
  • Directives for Mediapartners-Google
  • Directives for GoogleBot News
  • Directives for 008
  • Chat GPT
  • Sitemap
Back to top

Warnings

  • 2 invalid lines.
Back to top