Log In Sign Up

joinamble.com
robots.txt

Robots Exclusion Standard data for joinamble.com

Resource Scan

Scan Details

Site Domain joinamble.com
Base Domain joinamble.com
Scan Status Ok
Last Scan2026-02-06T16:52:56+00:00
Next Scan 2026-03-08T16:52:56+00:00

Last Scan

Scanned2026-02-06T16:52:56+00:00
URL https://joinamble.com/robots.txt
Redirect https://www.joinamble.com/robots.txt
Redirect Domain www.joinamble.com
Redirect Base joinamble.com
Domain IPs 75.2.70.75, 99.83.190.102
Redirect IPs 13.203.125.58, 13.233.175.166, 3.109.243.18
Response IP 54.238.67.66
Found Yes
Hash 40889cda77b06325f95bf74e1aefbdbb4b185f66255ef986cb0d0760c1576f5a
SimHash 5a1ed411ccae

Groups

gptbot

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

google-extended

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

meta-externalagent

Rule Path
Allow /

applebot-extended

Rule Path
Allow /

applebot

Rule Path
Allow /

bytespider

Rule Path
Allow /

*

Rule Path
Disallow /admin/
Disallow /private/
Disallow /wp-admin/
Disallow /api/
Disallow /.env
Disallow /config/
Back to top

Other Records

Field Value
sitemap https://www.joinamble.com/sitemap.xml
Back to top

Comments

  • Optimized robots.txt for AI Bot Accessibility
  • === HIGH PRIORITY AI BOTS (Recommended: Allow) ===
  • GPTBot - OpenAI's crawlers for ChatGPT training data and real-time search. OAI-SearchBot handles live web browsing, GPTBot for training data.
  • OAI-SearchBot - OpenAI's crawlers for ChatGPT training data and real-time search. OAI-SearchBot handles live web browsing, GPTBot for training data.
  • PerplexityBot - Perplexity AI's real-time web crawler that provides current information for AI answers. Blocking prevents your site from appearing in Perplexity search results.
  • Google-Extended - Google's crawler specifically for AI training data (Bard/Gemini). Separate from regular search indexing. Blocks AI training while preserving Google Search visibility.
  • === TRAINING & DATA COLLECTION BOTS ===
  • Allow these if you want your content used for AI model training
  • facebookexternalhit - Meta's crawler for link previews, content analysis, and Meta AI training. Used across Facebook, Instagram, WhatsApp, and Meta AI products.
  • meta-externalagent - Meta's crawler for link previews, content analysis, and Meta AI training. Used across Facebook, Instagram, WhatsApp, and Meta AI products.
  • Applebot-Extended - Apple's dedicated AI training crawler for Apple Intelligence. Separate from regular Applebot to allow selective AI training control.
  • Applebot - Apple's main crawler for Siri, Spotlight search, and general Apple services. Essential for Apple ecosystem discoverability.
  • Bytespider - ByteDance's web crawler for TikTok and international AI products. Replaces older Bytedance user-agent with current Bytespider.
  • === GENERAL OPTIMIZATIONS ===
  • Sitemap helps AI bots discover your content efficiently
  • Include both apex and www variants for maximum compatibility
  • Invalid URL provided - please enter a valid URL to generate sitemap entries
  • === COMMON EXCLUSIONS ===
  • Block admin and private areas for all bots
Back to top