Log In Sign Up

echo-online.de
robots.txt

Robots Exclusion Standard data for echo-online.de

Resource Scan

Scan Details

Site Domain echo-online.de
Base Domain echo-online.de
Scan Status Ok
Last Scan2024-05-21T19:05:06+00:00
Next Scan 2024-05-28T19:05:06+00:00

Last Scan

Scanned2024-05-21T19:05:06+00:00
URL https://echo-online.de/robots.txt
Redirect https://www.echo-online.de/robots.txt
Redirect Domain www.echo-online.de
Redirect Base echo-online.de
Domain IPs 75.2.84.18, 99.83.234.173
Response IP 18.156.157.251
Found Yes
Hash 37f509e5793e8da10691bd1df51f31001632a11d7894687232aa8794c067ae02
SimHash a83cd164cd74

Groups

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

*

Rule Path
Disallow /api/
Disallow /archive/
Disallow /na/
Disallow /service/profil/
Back to top

Other Records

Field Value
sitemap https://www.echo-online.de/index-sitemap.xml
Back to top

Comments

  • ChatGPT Plugins
  • Common Crawl
  • Bard
  • Meta’s bot that crawls public web pages to improve language models for their speech recognition technology
  • Used for several purposes, apparently also selling crawled data to LLM companies (http://omgili.com/crawler.html)
Back to top