Log In Sign Up

texashomes2percentrebate.com
robots.txt

Robots Exclusion Standard data for texashomes2percentrebate.com

Resource Scan

Scan Details

Site Domain texashomes2percentrebate.com
Base Domain texashomes2percentrebate.com
Scan Status Ok
Last Scan2026-02-13T21:52:04+00:00
Next Scan 2026-03-15T21:52:04+00:00

Last Scan

Scanned2026-02-13T21:52:04+00:00
URL https://texashomes2percentrebate.com/robots.txt
Domain IPs 104.26.14.57, 104.26.15.57, 172.67.69.225, 2606:4700:20::681a:e39, 2606:4700:20::681a:f39, 2606:4700:20::ac43:45e1
Response IP 172.67.69.225
Found Yes
Hash 4bcd256c5f3c1f87c1a2b1bf6de7cf8a45b8164b2c24bd99a0d19f6a530d5e50
SimHash 68365050e6e3

Groups

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

msnbot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

perplexity-user

Rule Path
Allow /

claude-user

Rule Path
Allow /

gptbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

ahrefssiteaudit

Rule Path
Allow /

amazonbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

*

Rule Path
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /wp-admin/admin-ajax.php

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://texashomes2percentrebate.com/sitemap_index.xml
Back to top

Comments

  • ==========================================
  • TexasHomes2PercentRebate.com robots.txt
  • Safe, AI-citation friendly, and secure
  • Updated: 2026-01-07
  • ==========================================
  • --------------------------
  • 1) Search engines (DO NOT BLOCK)
  • --------------------------
  • --------------------------
  • 2) AI retrieval / citations (CRITICAL - allow)
  • These bots control whether you can be surfaced and cited in AI answers
  • --------------------------
  • OpenAI - ChatGPT live search & citations
  • OpenAI - user-initiated fetches (Custom GPTs, follow-ups)
  • Perplexity - user-initiated retrieval (citations)
  • Anthropic (Claude) - user retrieval
  • --------------------------
  • 3) AI training bots (INTENTIONALLY BLOCKED)
  • Blocking training does NOT affect SEO or AI citations
  • --------------------------
  • OpenAI training crawler
  • Anthropic training crawler
  • Common Crawl dataset crawler
  • Google AI training usage (not regular Google search)
  • --------------------------
  • 4) Social & preview crawlers (ALLOW)
  • --------------------------
  • --------------------------
  • 5) Ahrefs (SEO auditing & crawling - EXPLICITLY ALLOWED)
  • --------------------------
  • --------------------------
  • 6) Block low-value / abusive crawlers
  • --------------------------
  • --------------------------
  • 7) Default rules for everything else
  • --------------------------
  • --------------------------
  • 8) Sitemap
  • --------------------------
Back to top