Log In Sign Up

booster.io
robots.txt

Robots Exclusion Standard data for booster.io

Resource Scan

Scan Details

Site Domain booster.io
Base Domain booster.io
Scan Status Ok
Last Scan2025-10-02T04:00:21+00:00
Next Scan 2025-11-01T04:00:21+00:00

Last Scan

Scanned2025-10-02T04:00:21+00:00
URL https://booster.io/robots.txt
Domain IPs 104.21.33.107, 172.67.189.210, 2606:4700:3034::6815:216b, 2606:4700:3037::ac43:bdd2
Response IP 172.67.189.210
Found Yes
Hash 13f9ccaa88bdea2074eafbac7f716ed66ee4bb00a8667a766af44c0cb340b0a1
SimHash 61f499032fd4

Groups

*

Rule Path Comment
Disallow /wp-admin/ Standard WordPress admin block
Disallow /wp-content/uploads/wpforms/ Your specific WPForms upload block
Disallow /search Your internal site search results
Disallow /forums/users/ Your forum user profiles
Allow /wp-admin/admin-ajax.php Standard WordPress AJAX allow

googlebot

Rule Path Comment
Disallow Allow full access

adsbot-google

Rule Path Comment
Disallow Allow if you run Google Ads

bingbot

Rule Path Comment
Disallow Allow full access

slurp

Rule Path
Disallow

duckduckbot

Rule Path
Disallow

gptbot

Rule Path Comment
Disallow Allowing OpenAI's web crawler

google-extended

Rule Path Comment
Disallow Allowing Google's AI user agent

claudebot

Product Comment
claudebot Anthropic's crawler (monitor for actual user agent)
Rule Path Comment
Disallow Allowing for now

perplexitybot

Rule Path Comment
Disallow Allowing Perplexity AI

youbot

Rule Path Comment
Disallow Allowing You.com

chatgpt-user

Product Comment
chatgpt-user User-initiated browsing via ChatGPT
Rule Path
Disallow

ccbot

Product Comment
ccbot Common Crawl
Rule Path Comment
Disallow / Consider disallowing to prevent use in broad, unattributed datasets

amazonbot

Rule Path Comment
Disallow / Consider disallowing if no benefit and high traffic/resource use
Back to top

Other Records

Field Value
sitemap https://booster.io/sitemap_index.xml
Back to top

Comments

  • General rules for all bots
  • Add any other general disallows for private areas, specific files, or query parameters if needed.
  • E.g., Disallow: /checkout/
  • E.g., Disallow: /my-account/
  • E.g., Disallow: /*?s= (Alternative for site search if /search isn't the only path)
  • --- Specific Search Engine Crawlers (Ensure broad access) ---
  • --- LLM & AI Crawlers - Permissive with Stated Terms (via /license page) ---
  • --- Potentially More Restrictive or Bots to Watch ---
  • Add other specific known problematic scrapers or LLM training bots you wish to block here.
  • Example:
  • User-agent: SomeNewAggressiveScraperBot
  • Disallow: /
  • --- Yoast SEO Sitemap ---
  • Yoast typically adds its sitemap reference. If it doesn't add it automatically
  • after these changes, or if you want to ensure it's present, you can keep it.
  • If Yoast re-adds its entire block, you might need to adjust this structure
  • or configure Yoast not to write to robots.txt if it causes conflicts.
  • For now, we'll assume Yoast's sitemap line is sufficient if it adds it.
  • If not, add your sitemap line here:
  • It's generally better to have only one User-agent: * block that is comprehensive,
  • or ensure more specific user-agent blocks appear AFTER a general User-agent: *
  • if you intend for them to be the primary rule for those specific bots.
  • The structure above prioritizes specific bot instructions.
Back to top

Warnings

  • 1 invalid line.
Back to top