Log In Sign Up

mitgurukul.com
robots.txt

Robots Exclusion Standard data for mitgurukul.com

Resource Scan

Scan Details

Site Domain mitgurukul.com
Base Domain mitgurukul.com
Scan Status Ok
Last Scan2026-02-09T05:24:23+00:00
Next Scan 2026-02-23T05:24:23+00:00

Last Scan

Scanned2026-02-09T05:24:23+00:00
URL https://mitgurukul.com/robots.txt
Domain IPs 2a02:4780:15:4a59:d491:aaad:b948:2cd4, 2a02:4780:38:256:555:316f:70ac:7b2b, 77.37.66.55, 93.127.201.124
Response IP 77.37.66.103
Found Yes
Hash 36b370f762e1b27fa3e9ddc6c0d95f507d58e7a365efdd1fb526513bb5630338
SimHash 7196d2206517

Groups

*

Rule Path
Disallow /wp-admin/admin-ajax.php
Allow /

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

yandex

Rule Path
Allow /

baiduspider

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

applebot

Rule Path
Allow /

gptbot

Rule Path
Allow /

claudebot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

google-extended

Rule Path
Allow /

bingai

Rule Path
Allow /

ccbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://mitgurukul.com/sitemap.xml
Back to top

Comments

  • Major Search Engine Bots
  • Google Search
  • Bing Search
  • Yahoo Search
  • Yandex Search
  • Baidu Search
  • DuckDuckGo Bot
  • Apple Search Bot
  • Major AI Bots
  • OpenAI GPTBot (used for ChatGPT / GPT training)
  • Anthropic ClaudeBot
  • Perplexity AI Bot
  • Google AI Crawler (Gemini/Bard training)
  • Microsoft AI Bot (Bing AI / Copilot)
  • Common Crawl (used by many AI models)
Back to top