Log In Sign Up

dramsch.net
robots.txt

Robots Exclusion Standard data for dramsch.net

Resource Scan

Scan Details

Site Domain dramsch.net
Base Domain dramsch.net
Scan Status Ok
Last Scan2026-02-24T02:03:41+00:00
Next Scan 2026-03-03T02:03:41+00:00

Last Scan

Scanned2026-02-24T02:03:41+00:00
URL https://dramsch.net/robots.txt
Domain IPs 104.21.74.163, 172.67.204.67, 2606:4700:3034::ac43:cc43, 2606:4700:3035::6815:4aa3
Response IP 172.67.204.67
Found Yes
Hash b554de3729efac580e7eb611732467dcbcb3b898b9966ad3f020731dbc370135
SimHash 701a5950f5c0

Groups

googlebot
bingbot
slurp
duckduckbot
baiduspider
yandexbot
ahrefsbot
semrushbot
mj12bot
dotbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

facebookbot
facebookexternalhit
meta-externalagent
linkedinbot
whatsapp
twitterbot
twitterbot
pinterest
pinterestbot

Rule Path
Allow /

gptbot
chatgpt-user
ccbot
anthropic-ai
claude-web
claudebot
cohere-ai
bytespider
img2dataset
scrapy
timpibot
velenpublicwebcrawler
youbot
petalbot
omgili
omgilibot
diffbot
apify
dataforseobot

Rule Path
Disallow /

google-extended
googleother
googleother-image
googleother-video
applebot-extended
perplexity-ai
perplexitybot
oai-searchbot
amazonbot

Rule Path
Allow /

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

seokicks
alexibot
surveybot
xenu's
xenu's link sleuth 1.1c
rogerbot
psbot
turnitinbot
clickagy
nutch
blexbot
ezooms
majestic-12
majestic-seo
dsearch
megaindex
blekkobot
nerdybot
jamesbot
tineye
tineye-bot
konqueror
searchmetricsbot
seznambot
extlinksbot
mail.ru_bot
spbot
linkpadbot
bdcbot
updownerbot
voilabot
zumbot
youdaobot
apitool
thumbnail.cz robot
mj12bot
heritrix
copyrightcheck
aboundex
bytedance
bytedance spider
magpie-crawler

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://dramsch.net/sitemapindex.xml
Back to top

Comments

  • robots.txt for https://dramsch.net
  • Last updated: 2025
  • Primary sitemap reference
  • ==========================================
  • Search Engine Crawlers - Full Access
  • ==========================================
  • ==========================================
  • Social Media Crawlers - Full Access
  • ==========================================
  • ==========================================
  • AI Training Scrapers - Blocked
  • ==========================================
  • ==========================================
  • Search Engine AI Features - Allowed
  • ==========================================
  • ==========================================
  • All Other Crawlers - Standard Rules
  • ==========================================
  • Block private/temporary pages
  • Block duplicate content paths
  • Block search and filter pages
  • Allow important assets
  • ==========================================
  • Crawl Rate Recommendations
  • ==========================================
  • Respect crawl delays for non-critical bots
  • ==========================================
  • Additional Directives
  • ==========================================
  • Specify the preferred host (without www)
  • Clean URLs - help crawlers understand URL structure
  • Request indexing of important sections
  • ==========================================
  • Bad Bots - Complete Block
  • ==========================================
  • ==========================================
  • End of robots.txt
  • ==========================================
Back to top

Warnings

  • 37 invalid lines.
  • `clean-param` is not a known field.
  • `host` is not a known field.
Back to top