Log In Sign Up

aitrackerhive.com
robots.txt

Robots Exclusion Standard data for aitrackerhive.com

Resource Scan

Scan Details

Site Domain aitrackerhive.com
Base Domain aitrackerhive.com
Scan Status Ok
Last Scan2026-03-18T05:22:44+00:00
Next Scan 2026-03-25T05:22:44+00:00

Last Scan

Scanned2026-03-18T05:22:44+00:00
URL https://aitrackerhive.com/robots.txt
Domain IPs 104.21.94.115, 172.67.223.64, 2606:4700:3030::6815:5e73, 2606:4700:3033::ac43:df40
Response IP 172.67.223.64
Found Yes
Hash 5af4f2e9e4bd4c743b78489b58f7bf08a8b2380266efe2ad82199ad43f7403b6
SimHash 4ec2bb51c5cf

Groups

*

Rule Path
Disallow /api/
Disallow /cdn-cgi/
Disallow /_next/
Disallow /sign-in
Disallow /sign-up
Disallow /*/sign-in
Disallow /*/sign-up
Disallow /admin
Disallow /*/admin
Disallow /settings
Disallow /*/settings
Disallow /chat
Disallow /*/chat
Disallow /activity
Disallow /*/activity
Disallow /likes
Disallow /*/likes
Disallow /history
Disallow /*/history
Disallow /blog
Disallow /*/blog
Disallow /pricing
Disallow /*/pricing
Disallow /showcases
Disallow /*/showcases
Disallow /docs
Disallow /*/docs
Disallow /ai-image-generator
Disallow /ai-music-generator
Disallow /ai-video-generator
Allow /ads.txt

mediapartners-google

Rule Path
Allow /
Allow /ads.txt

googlebot

Rule Path
Allow /
Disallow /api/
Disallow /cdn-cgi/
Disallow /_next/
Disallow /sign-in
Disallow /sign-up
Disallow /*/sign-in
Disallow /*/sign-up
Disallow /admin
Disallow /*/admin
Disallow /settings
Disallow /*/settings
Disallow /chat
Disallow /*/chat
Disallow /activity
Disallow /*/activity
Disallow /likes
Disallow /*/likes
Disallow /history
Disallow /*/history
Disallow /blog
Disallow /*/blog
Disallow /pricing
Disallow /*/pricing
Disallow /showcases
Disallow /*/showcases
Disallow /docs
Disallow /*/docs

gptbot
chatgpt-user
claude-web
anthropic-ai
anthropic-ai
perplexitybot
googleother
duckassistbot
ccbot

Rule Path
Allow /
Allow /artists
Allow /*/artists
Disallow /api/
Disallow /admin
Disallow /*/admin
Disallow /settings
Disallow /*/settings
Disallow /chat
Disallow /*/chat
Disallow /activity
Disallow /*/activity
Disallow /sign-in
Disallow /sign-up
Disallow /*/sign-in
Disallow /*/sign-up

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

sogou spider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

sosospider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

youdaobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

yetibot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

seznambot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

rdfbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2
Back to top

Other Records

Field Value
sitemap https://aitrackerhive.com/sitemap-index.xml
Back to top

Comments

  • =============================================================================
  • AI Tracker Hive - robots.txt
  • Enterprise-grade crawler directives
  • Last updated: 2025-12-24
  • =============================================================================
  • -----------------------------------------------------------------------------
  • Default Rules (All Crawlers)
  • -----------------------------------------------------------------------------
  • System and Infrastructure (Never index)
  • Authentication Pages (All languages)
  • Admin Dashboard (All languages)
  • User Settings (All languages)
  • User-Specific Content (All languages)
  • Blog (Not implemented - temporary)
  • Pricing (Template page - temporary)
  • Deleted/Legacy Pages (Prevent crawling old links)
  • Allow ads.txt for monetization verification
  • -----------------------------------------------------------------------------
  • Google AdSense Bot
  • -----------------------------------------------------------------------------
  • -----------------------------------------------------------------------------
  • Googlebot (Standard)
  • -----------------------------------------------------------------------------
  • -----------------------------------------------------------------------------
  • AI Crawlers (LLM Training & Retrieval)
  • -----------------------------------------------------------------------------
  • Allow AI crawlers to access main content
  • Block private/system paths
  • LLM-specific content guides (non-standard but useful)
  • These files provide structured content for AI understanding
  • LLM-Content: https://aitrackerhive.com/llms.txt
  • LLM-Full-Content: https://aitrackerhive.com/llms-full.txt
  • -----------------------------------------------------------------------------
  • Rate-Limited Crawlers
  • Polite crawl delays to prevent server overload
  • -----------------------------------------------------------------------------
  • -----------------------------------------------------------------------------
  • Sitemaps
  • -----------------------------------------------------------------------------
Back to top