jivansutra.com
robots.txt

Robots Exclusion Standard data for jivansutra.com

Resource Scan

Scan Details

Site Domain jivansutra.com
Base Domain jivansutra.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2026-01-19T04:45:18+00:00
Next Scan 2026-01-26T04:45:18+00:00

Last Successful Scan

Scanned2026-01-11T04:45:11+00:00
URL https://jivansutra.com/robots.txt
Domain IPs 2a02:4780:27:1525:0:18eb:6f28:3, 91.108.101.77
Response IP 91.108.101.77
Found Yes
Hash 1a1dac339789cf9410bb946f246507cd9576f91d83f7fa2f2ce1531412b946fd
SimHash 778acb4084b0

Groups

*

Rule Path
Disallow /xmlrpc.php
Disallow /readme.html
Disallow /trackback/
Disallow /refer/
Disallow /feed/
Disallow /comments/feed/
Disallow /author/
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /wp-json/wp/v2/users/
Disallow /wp-content/plugins/
Disallow /wp-content/cache/
Disallow /wp-content/themes/
Disallow /cgi-bin/
Disallow /*.sql$
Disallow /*.zip$
Disallow /*.tar$
Disallow /*.gz$
Allow /wp-admin/admin-ajax.php
Allow /wp-content/uploads/
Allow /wp-content/
Allow /wp-includes/
Allow /wp-json/

ai2bot
amazonbot
anthropic-ai
applebot
applebot-extended
barkrowler
blexbot
bytespider
cazoodlebot
ccbot
chatgpt-user
claude-web
claudebot
cohere-ai
deepseek
deepseek-ai-crawler
deepseekbot
deepseekcrawler
deepseekindexer
deepseek-webagent
imagebot-deepseek
diffbot
duckassistbot
exaleadbot
facebookbot
gensparkcrawler
genspark-ai-bot
genspark-dataminer
gensparkai-indexer
genspark-ai-crawler
gigabot
google-extended
gptbot
imagesiftbot
linkedinbot
meta-externalagent
meta-externalfetcher
mistralai
omgilibot
omgili
perplexitybot
perplexity-user
petalbot
riddler
slackbot-linkexpanding
sogou spider
surveybot
timpibot
webzio-extended
youbot
zoominfobot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.jivansutra.com/sitemap_index.xml

Comments

  • =========================
  • Jivansutra.com Robots.txt
  • =========================
  • -------------------------
  • Block AI Crawlers & Scrapers
  • -------------------------
  • -------------------------
  • Sitemap for Full Indexing
  • -------------------------