Log In Sign Up

abhishekschauhan.com
robots.txt

Robots Exclusion Standard data for abhishekschauhan.com

Resource Scan

Scan Details

Site Domain abhishekschauhan.com
Base Domain abhishekschauhan.com
Scan Status Ok
Last Scan2026-02-12T16:46:05+00:00
Next Scan 2026-02-19T16:46:05+00:00

Last Scan

Scanned2026-02-12T16:46:05+00:00
URL https://abhishekschauhan.com/robots.txt
Domain IPs 162.214.80.40
Response IP 162.214.80.40
Found Yes
Hash 56ec22a6d4fd4bb13fb5c4e3184cba545c95bca0888d8266b3bcf148276d5c6e
SimHash 7d0583510d72

Groups

*

Rule Path
Disallow

gptbot

Product Comment
gptbot OpenAI (ChatGPT)
Rule Path
Disallow

chatgpt-user

Product Comment
chatgpt-user ChatGPT browser plugin crawler
Rule Path
Disallow

google-extended

Product Comment
google-extended Google Bard / Gemini AI
Rule Path
Disallow

googleother

Product Comment
googleother Google indexing for other AI tools
Rule Path
Disallow

bingbot

Product Comment
bingbot Bing Search + Copilot
Rule Path
Disallow

bingpreview

Product Comment
bingpreview Bing Copilot previews
Rule Path
Disallow

msnbot

Product Comment
msnbot Microsoft bot
Rule Path
Disallow

facebookbot

Product Comment
facebookbot Facebook AI data
Rule Path
Disallow

claudebot

Product Comment
claudebot Anthropic AI
Rule Path
Disallow

anthropic-ai

Product Comment
anthropic-ai Anthropic alternate crawler ID
Rule Path
Disallow

ccbot

Product Comment
ccbot Common Crawl
Rule Path
Disallow

perplexitybot

Product Comment
perplexitybot Perplexity AI
Rule Path
Disallow

amazonbot

Product Comment
amazonbot Amazon Search + AI
Rule Path
Disallow

applebot

Product Comment
applebot Apple Search & Siri
Rule Path
Disallow

duckduckbot

Product Comment
duckduckbot DuckDuckGo + AI features
Rule Path
Disallow

youbot

Product Comment
youbot You.com AI Search
Rule Path
Disallow

cohere-ai

Product Comment
cohere-ai Cohere AI models
Rule Path
Disallow

imagesiftbot

Product Comment
imagesiftbot AI image search
Rule Path
Disallow

diffbot

Product Comment
diffbot AI web extraction
Rule Path
Disallow

baiduspider

Product Comment
baiduspider Baidu AI Search
Rule Path
Disallow

yandexbot

Product Comment
yandexbot Yandex AI Search
Rule Path
Disallow

neevabot

Product Comment
neevabot Neeva AI Search
Rule Path
Disallow

dataforseobot

Product Comment
dataforseobot SEO & AI tools
Rule Path
Disallow

magpie-crawler

Product Comment
magpie-crawler AI research crawler
Rule Path
Disallow

seomoz

Product Comment
seomoz Moz crawler
Rule Path
Disallow

semrushbot

Product Comment
semrushbot Semrush SEO & AI tools
Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://www.abhishekschauhan.com/sitemap.xml
sitemap https://www.abhishekschauhan.com/sitemap-home.xml
sitemap https://www.abhishekschauhan.com/sitemap-posts.xml
sitemap https://www.abhishekschauhan.com/sitemap-pages.xml
sitemap https://www.abhishekschauhan.com/sitemap-categories.xml
sitemap https://www.abhishekschauhan.com/sitemap-tags.xml
sitemap https://www.abhishekschauhan.com/sitemap-archives.xml
sitemap https://www.abhishekschauhan.com/sitemap-custom-taxonomies.xml
sitemap https://www.abhishekschauhan.com/sitemap-custom-posts.xml
sitemap https://www.abhishekschauhan.com/sitemap-attachment.xml
sitemap https://www.abhishekschauhan.com/sitemap-posts.xml
Back to top

Comments

  • robots.txt for abhishekschauhan.com
  • Goal: Allow ALL search engines, AI crawlers, and generative answer engines
  • Sitemap location
  • Allow everything for everyone
  • --- Major AI & Generative Search Crawlers ---
Back to top