Log In Sign Up

dwds.de
robots.txt

Robots Exclusion Standard data for dwds.de

Resource Scan

Scan Details

Site Domain dwds.de
Base Domain dwds.de
Scan Status Ok
Last Scan2024-09-20T21:11:29+00:00
Next Scan 2024-10-20T21:11:29+00:00

Last Scan

Scanned2024-09-20T21:11:29+00:00
URL https://dwds.de/robots.txt
Redirect https://www.dwds.de/robots.txt
Redirect Domain www.dwds.de
Redirect Base dwds.de
Domain IPs 194.95.188.49
Redirect IPs 194.95.188.49
Response IP 194.95.188.49
Found Yes
Hash ca61b6bb24f0a9a6dee836679c158aa5f6b3a6709d5bdea95ddf438f77a18d6f
SimHash 22581f568df7

Groups

*

Rule Path
Disallow /fussball
Disallow /freq
Disallow /profile
Disallow /r
Disallow /wp

anthropic-ai

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bloodhound

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

cydralspider

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

downloadexpress

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

gammaspider

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

meltwater

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

objectssearch

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

pimptrain

Rule Path
Disallow /

raven

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

wapspider

Rule Path
Disallow /

webzinger

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.dwds.de/sitemap.xml
Back to top

Comments

  • text and data mining bots
  • Common Crawl
  • ChatGPT users, see also below ChatGPT
  • Cydral Image Search
  • ChatGPT
  • MeltwaterNews
  • Meta
  • webz.io
  • Legal Notice: The Digital Dictionary of the German Language (DWDS, dwds.de)
  • explicitly retains the right to utilize its content for commercial text and
  • data mining in accordance with § 44b UrhG. Unauthorized use of robots or other
  • automated mechanisms to access dwds.de or to gather or mine data is strictly
  • forbidden without explicit consent from DWDS. DWDS may, at its discretion,
  • allow specific automated access to designated dwds.de pages. To request
  • permission for crawling dwds.de, data collection, or usage, please contact
  • <mailto:dwds@bbaw.de>.
Back to top