Log In Sign Up

ditstek.com
robots.txt

Robots Exclusion Standard data for ditstek.com

Resource Scan

Scan Details

Site Domain ditstek.com
Base Domain ditstek.com
Scan Status Ok
Last Scan2025-09-24T19:01:40+00:00
Next Scan 2025-10-24T19:01:40+00:00

Last Scan

Scanned2025-09-24T19:01:40+00:00
URL https://ditstek.com/robots.txt
Redirect https://www.ditstek.com/robots.txt
Redirect Domain www.ditstek.com
Redirect Base ditstek.com
Domain IPs 107.180.11.176
Redirect IPs 107.180.11.176
Response IP 107.180.11.176
Found Yes
Hash be6b23b886c443be7b72ae828160b3103b2979558d3f574da74632f9a2a7a179
SimHash d99d12418553

Groups

*

Rule Path
Allow /
Disallow /thankyou

gptbot

Rule Path
Allow /

ccbot

Rule Path
Allow /

google-extended

Rule Path
Allow /

googlebot

Rule Path
Allow /

deepmind-useragent

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

claudebot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

metabot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

applebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

youbot

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

mistral-ai

Rule Path
Allow /

alephalphabot

Rule Path
Allow /

neevabot

Rule Path
Allow /

ia_archiver

Rule Path
Allow /

yandex

Rule Path
Allow /

inflectionbot

Rule Path
Allow /

serpapibot

Rule Path
Allow /

sogou web spider

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://ditstek.com/sitemap.xml
Back to top

Comments

  • OpenAI
  • Common Crawl
  • Google AI
  • Googlebot (used in AI features like Bard/Gemini)
  • DeepMind
  • Anthropic
  • ClaudeBot
  • Perplexity AI
  • Meta AI
  • Amazon AI
  • Apple Siri / Applebot
  • Microsoft Bing AI
  • DuckDuckGo
  • You.com
  • Cohere AI
  • Mistral AI
  • Aleph Alpha
  • Neeva (Snowflake)
  • Archive.org
  • Yandex
  • Inflection AI
  • SerpAPI
  • Sogou
  • Sitemap
Back to top