Log In Sign Up

couponscodehut.com
robots.txt

Robots Exclusion Standard data for couponscodehut.com

Resource Scan

Scan Details

Site Domain couponscodehut.com
Base Domain couponscodehut.com
Scan Status Ok
Last Scan2026-02-04T20:58:07+00:00
Next Scan 2026-02-11T20:58:07+00:00

Last Scan

Scanned2026-02-04T20:58:07+00:00
URL https://couponscodehut.com/robots.txt
Domain IPs 2a02:4780:84:6644:5a5e:e878:e1dd:ece4, 2a02:4780:84:980d:b37b:fee:c316:b739, 84.32.84.103, 84.32.84.251
Response IP 77.37.115.167
Found Yes
Hash 5d939ccaf19e94aaea6154c0b0c49d1aa465ee95106bfcb13f126e648a4fef43
SimHash 65b39b2165f4

Groups

*

Rule Path
Allow /
Allow /llms.txt
Allow /llms-full.txt
Allow /llms.php
Allow /stores/
Allow /categories/
Allow /blog/
Allow /page/
Allow /coupons
Allow /search
Disallow /admin/
Disallow /admin
Disallow /config/
Disallow /config
Disallow /includes/
Disallow /includes
Disallow /database/
Disallow /database
Disallow /ajax/
Disallow /ajax
Disallow /*?sort=
Disallow /*?search=
Disallow /*?letter=
Allow /*?page=
Disallow /*.sql$
Disallow /*.log$
Disallow /*.bak$
Disallow /uploads/banners/
Disallow /uploads/stores/placeholder*

googlebot

Rule Path
Allow /
Disallow /admin/
Disallow /config/
Disallow /includes/
Disallow /database/
Disallow /ajax/

googlebot-image

Rule Path
Allow /uploads/
Allow /assets/
Disallow /admin/

bingbot

Rule Path
Allow /
Disallow /admin/
Disallow /config/
Disallow /includes/
Disallow /database/
Disallow /ajax/

gptbot

Rule Path
Allow /
Allow /llms.txt
Allow /llms-full.txt
Allow /llms.php
Disallow /admin/
Disallow /config/
Disallow /includes/
Disallow /database/

chatgpt-user

Rule Path
Allow /
Allow /llms.txt
Allow /llms-full.txt
Allow /llms.php
Disallow /admin/
Disallow /config/

google-extended

Rule Path
Allow /
Allow /llms.txt
Allow /llms-full.txt
Disallow /admin/
Disallow /config/

anthropic-ai

Rule Path
Allow /
Allow /llms.txt
Allow /llms-full.txt
Allow /llms.php
Disallow /admin/
Disallow /config/

claude-web

Rule Path
Allow /
Allow /llms.txt
Allow /llms-full.txt
Allow /llms.php
Disallow /admin/
Disallow /config/

perplexitybot

Rule Path
Allow /
Allow /llms.txt
Allow /llms-full.txt
Disallow /admin/
Disallow /config/

cohere-ai

Rule Path
Allow /
Allow /llms.txt
Allow /llms-full.txt
Disallow /admin/
Disallow /config/

facebookbot

Rule Path
Allow /
Allow /llms.txt
Disallow /admin/
Disallow /config/

meta-externalagent

Rule Path
Allow /
Allow /llms.txt
Allow /llms-full.txt
Disallow /admin/
Disallow /config/

ccbot

Rule Path
Allow /
Allow /llms.txt
Allow /llms-full.txt
Disallow /admin/
Disallow /config/
Disallow /database/
Back to top

Other Records

Field Value
sitemap https://couponscodehut.com/sitemap.xml
Back to top

Comments

  • ============================================
  • Robots.txt for CouponsCodeHut.com
  • Technical SEO Configuration
  • Last Updated: 2026-01-02
  • ============================================
  • Default rules for all crawlers
  • Allow all public-facing pages
  • AI/LLM Discovery Files
  • Block admin and internal routes
  • Block query parameters that create duplicate content
  • Allow pagination (important for crawling all content)
  • Block specific file types
  • Block uploads subdirectories that shouldn't be indexed directly
  • ============================================
  • Specific crawler rules
  • ============================================
  • Google
  • Google Images
  • Bing
  • ============================================
  • AI Crawlers and LLM Agents
  • These crawlers are used by AI companies to index content
  • ============================================
  • OpenAI GPTBot
  • OpenAI ChatGPT-User (browsing mode)
  • Google Bard / Gemini
  • Anthropic Claude
  • Perplexity AI
  • Cohere AI
  • Meta AI
  • Common Crawl (used by many AI training datasets)
  • ============================================
  • Crawl-delay settings (optional - be careful)
  • Uncomment if experiencing high server load
  • ============================================
  • User-agent: *
  • Crawl-delay: 1
  • ============================================
  • Sitemap location
  • ============================================
  • ============================================
  • AI/LLM Discovery Files
  • ============================================
  • LLMs.txt - AI-friendly documentation following llmstxt.org standard
  • Static version: /llms.txt
  • Extended version: /llms-full.txt
  • Dynamic version with live data: /llms.php
Back to top