Log In Sign Up

danesh.land
robots.txt

Robots Exclusion Standard data for danesh.land

Resource Scan

Scan Details

Site Domain danesh.land
Base Domain danesh.land
Scan Status Ok
Last Scan2025-12-02T13:52:37+00:00
Next Scan 2026-01-01T13:52:37+00:00

Last Scan

Scanned2025-12-02T13:52:37+00:00
URL https://danesh.land/robots.txt
Domain IPs 88.135.68.26
Response IP 88.135.68.26
Found Yes
Hash 3093fa2a850f1db1f368319bdfaea71f3bd53d363b08a3ec87994a5ddf0dbdbf
SimHash 6371294b25ac

Groups

*
googlebot
googlebot-image
googlebot-news
googlebot-video
bingbot
duckduckbot
yandex
baiduspider
applebot
sogou
exabot
facebot
pinterestbot
openai
gptbot
google-extended
google-deepmind
gemini
anthropic
claudebot
perplexity
mistral
cohere
ai21
youbot
poebot
jasper
copyai
writerbot
huggingface
stabilityai
deepseek
grok
xai
metaai
replika
chatgpt-user
characterai
llama
claude
commandr
alephalpha
togetherai
vercelai
anyscale
amazon-bedrock
ollama
inflectionai
gigachat
erniebot

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Allow /wp-includes/js/
Allow /wp-includes/css/
Allow /wp-content/plugins/
Allow /wp-content/themes/
Allow /wp-content/uploads/
Allow /wp-json/
Allow /?rest_route=*
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /wp-includes/
Disallow /readme.html
Disallow /license.txt
Disallow /xmlrpc.php
Disallow /cgi-bin/
Disallow /cgi-sys/
Disallow /tmp/
Disallow /private/
Disallow /backup/
Disallow /error_log
Disallow /.git/
Disallow /.svn/
Disallow /.env
Disallow /hidden/
Disallow /cache/
Disallow /test/
Disallow /beta/
Disallow /sandbox/
Disallow /staging/
Disallow /cart/
Disallow /cart*
Disallow /checkout/
Disallow /checkout*
Disallow /my-account/
Disallow /wishlist/
Disallow /compare/
Disallow /*add-to-cart*
Disallow /signup/
Disallow /signin/
Disallow /login/
Disallow /logout/
Disallow /thank-you/
Disallow /order-received/
Disallow /order-tracking/
Disallow /account-settings/
Disallow *?*
Disallow */?*
Disallow *%D8%9F*
Disallow /?v*
Disallow /?s=
Disallow /search/
Disallow *?orderby*
Disallow *?filter*
Disallow *?add-to-cart*
Disallow *?utm_*
Disallow *?ref=*
Disallow *?preview=*
Disallow *?replytocom*
Disallow *?share=*
Disallow *?fbclid*
Disallow *?gclid*
Disallow *?page=*
Disallow *?ver=*
Disallow *?amp=*
Disallow *?lang=*
Disallow /feed/
Disallow */feed/
Disallow /comments/feed/
Disallow /archive/
Disallow /category/*/feed/
Disallow /*/amp/

gptbot

Rule Path
Allow /

google-extended

Rule Path
Allow /

ccbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://danesh.land/sitemap-index.xml
sitemap https://danesh.land/video-sitemap-index.xml
sitemap https://danesh.land/image-sitemap-index.xml
sitemap https://danesh.land/audio-sitemap-index.xml
sitemap https://danesh.land/custom-html-sitemap.html
sitemap https://danesh.land/sitemap-index.xml
Back to top

Comments

  • Robots.txt for https://danesh.land
  • AI + SEO Optimized (WordPress + WooCommerce)
  • Version: 2025-10-15 — Author: Ali ATR
  • ------------------------------
  • 1️⃣ General Access Permissions
  • ------------------------------
  • ------------------------------
  • 2️⃣ AI & Large Language Model (LLM) Crawlers
  • ------------------------------
  • Allow essential crawling
  • ------------------------------
  • 3️⃣ Allow Important WP Resources
  • ------------------------------
  • ------------------------------
  • 4️⃣ Disallow Sensitive or Internal Paths
  • ------------------------------
  • ------------------------------
  • 5️⃣ Block eCommerce Private Areas
  • ------------------------------
  • ------------------------------
  • 6️⃣ Block URL Parameters & Duplicate Variants
  • ------------------------------
  • ------------------------------
  • 7️⃣ Block Feeds, Archives, Tags & Pagination
  • ------------------------------
  • ------------------------------
  • 8️⃣ Crawl-Delay (Optional)
  • ------------------------------
  • Helps prevent server overload from aggressive crawlers
  • Crawl-delay: 7
  • ------------------------------
  • 9️⃣ Sitemaps
  • ------------------------------
  • ------------------------------
  • 🔟 AI & Data Ethics Declaration
  • ------------------------------
  • AI crawlers listed above are authorized to index and summarize public content.
  • Redistribution, dataset training of paid content, or unauthorized copying is prohibited.
  • All crawling must respect ethical usage and data ownership standards.
  • ✅ Final Notes:
  • - Fully compatible with Google SGE, Gemini, GPTBot, Grok, and Perplexity.
  • - Blocks thin, duplicate, session, and private pages for cleaner SEO.
  • - Ensures visibility for AI Search & LLM understanding.
  • - Safe for high-traffic WooCommerce + Elementor websites.
  • START Smart Sitemap Generator
  • END Smart Sitemap Generator
Back to top