Log In Sign Up

fridge.com
robots.txt

Robots Exclusion Standard data for fridge.com

Resource Scan

Scan Details

Site Domain fridge.com
Base Domain fridge.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-02-06T05:09:32+00:00
Next Scan 2026-02-20T05:09:32+00:00

Last Successful Scan

Scanned2026-01-22T05:04:28+00:00
URL https://fridge.com/robots.txt
Domain IPs 216.150.1.1
Response IP 216.150.1.1
Found Yes
Hash 30b600d9a4b01409b4c658d88c27e882120b634c573c1e92ea3b34ab194dfdb3
SimHash 6d046753c4f5

Groups

*

Rule Path
Allow /
Disallow /api/
Disallow /admin-panel/
Disallow /_next/
Disallow /auth/
Disallow /signin
Disallow /signup
Disallow /docs/
Disallow /r/
Back to top

Other Records

Field Value
sitemap https://fridge.com/sitemap.xml
Back to top

Comments

  • Fridge.com Robots.txt
  • Last updated: December 27, 2025
  • Site is LIVE and indexed
  • ==============================================
  • UNIVERSAL DISALLOW RULES (Apply to ALL bots)
  • ==============================================
  • Note: In robots.txt, each User-agent block is independent.
  • We must repeat Disallow rules in each block, OR use User-agent: * first.
  • We use User-agent: * with Disallows, then specific bots inherit by matching *.
  • ==============================================
  • AI CRAWLERS — EXPLICITLY WELCOME
  • (All 7 Major LLMs + More)
  • ==============================================
  • These bots match User-agent: * above, which allows / but blocks sensitive paths.
  • The comments below document which AI crawlers we welcome.
  • OpenAI (ChatGPT): GPTBot, ChatGPT-User
  • Google AI (Gemini): Google-Extended
  • Anthropic (Claude): ClaudeBot, Claude-Web
  • Perplexity: PerplexityBot
  • Microsoft (Copilot): Bingbot
  • Meta (Llama): Meta-ExternalAgent, FacebookBot
  • xAI (Grok): (uses standard web crawling)
  • Common Crawl: CCBot
  • Cohere: cohere-ai
  • You.com: YouBot
  • ==============================================
  • SEARCH ENGINE CRAWLERS — WELCOME
  • ==============================================
  • Googlebot, Bingbot, DuckDuckBot, Slurp, Yandex, Applebot
  • All match User-agent: * above
  • ==============================================
  • SITEMAPS
  • ==============================================
  • ==============================================
  • LLM-SPECIFIC FILES
  • ==============================================
  • For AI assistants: comprehensive site information
  • ChatGPT, Perplexity, Gemini, Copilot, Claude, Llama, Grok, and more
  • See: https://fridge.com/llms.txt
Back to top