Log In Sign Up

vedicmarga.com
robots.txt

Robots Exclusion Standard data for vedicmarga.com

Resource Scan

Scan Details

Site Domain vedicmarga.com
Base Domain vedicmarga.com
Scan Status Ok
Last Scan2025-06-03T03:54:36+00:00
Next Scan 2025-06-10T03:54:36+00:00

Last Scan

Scanned2025-06-03T03:54:36+00:00
URL https://vedicmarga.com/robots.txt
Domain IPs 194.1.147.28, 194.1.147.92
Response IP 194.1.147.28
Found Yes
Hash 1678fc4b7b80d09ba89165b5998a7519bd9b2f54d06000521f5d8d7a808d1bce
SimHash 682cc84324a2

Groups

*

Rule Path Comment
Disallow /wp-admin/ -
Allow /wp-admin/admin-ajax.php -
Disallow /private/ Add any private directories or pages

Other Records

Field Value Comment
crawl-delay 10 Optional: Slows down bots to reduce server load

oai-searchbot
perplexitybot
andibot
youbot
phindbot
exabot
firecrawlagent

Product Comment
oai-searchbot OpenAI's search bot for ChatGPT
perplexitybot Perplexity AI search
andibot Andi AI search
youbot You.com AI search
phindbot Phind AI search
exabot Exa AI search
firecrawlagent Firecrawl for AI agents
Rule Path Comment
Allow / Allow these bots to crawl entire site
Disallow /private/ Block sensitive areas

googlebot
bingbot
slurp

Product Comment
slurp Yahoo's bot
Rule Path
Allow /
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /private/

facebookexternalhit

Rule Path
Allow /

gptbot
claudebot
anthropic-ai
google-extended
ccbot
bytespider
cohere-ai
omgili

Product Comment
gptbot OpenAI training bot
claudebot Anthropic training bot
anthropic-ai Anthropic general bot
google-extended Google AI training bot
ccbot Common Crawl, used for AI training
bytespider ByteDance AI training
cohere-ai Cohere AI training
omgili Omgili AI training
Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://vedicmarga.com/sitemap_index.xml
Back to top

Comments

  • General rules for all bots
  • Allow AI search/indexing bots for visibility
  • Allow traditional search engines
  • Allow social media previews
  • Block AI training bots to protect proprietary content
  • Sitemap for all crawlers
Back to top