foodgawker.vercel.app
robots.txt

Robots Exclusion Standard data for foodgawker.vercel.app

Resource Scan

Scan Details

Site Domain foodgawker.vercel.app
Base Domain foodgawker.vercel.app
Scan Status Ok
Last Scan2025-07-09T08:55:09+00:00
Next Scan 2025-08-08T08:55:09+00:00

Last Scan

Scanned2025-07-09T08:55:09+00:00
URL https://foodgawker.vercel.app/robots.txt
Domain IPs 216.198.79.195, 64.29.17.195
Response IP 64.29.17.67
Found Yes
Hash ee2cb3056ba3401165bb3a8506d9d4041529e63dbdc85e2e5e26e2520fa75c43
SimHash 2a048c14d961

Groups

*

Rule Path
Allow /
Allow /*/opengraph-image
Allow /*/twitter-image
Allow /about/opengraph-image

facebookexternalhit/*

Rule Path
Allow /

twitterbot

Rule Path
Allow /

whatsapp

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

googlebot

Rule Path
Allow /

Other Records

Field Value
sitemap https://foodgawker.vercel.app/sitemap.xml

Comments

  • Explicitly allow OpenGraph images
  • Allow common social media crawlers