Log In Sign Up

golem.de
robots.txt

Robots Exclusion Standard data for golem.de

Resource Scan

Scan Details

Site Domain golem.de
Base Domain golem.de
Scan Status Ok
Last Scan2024-11-09T10:25:25+00:00
Next Scan 2024-11-16T10:25:25+00:00

Last Scan

Scanned2024-11-09T10:25:25+00:00
URL https://golem.de/robots.txt
Redirect https://www.golem.de/robots.txt
Redirect Domain www.golem.de
Redirect Base golem.de
Domain IPs 2a00:13c8:f5::f:4b3d:148, 77.247.84.129
Redirect IPs 2a00:13c8:f5::f:4b3d:148, 77.247.84.129
Response IP 77.247.84.129
Found Yes
Hash fc460ba52b2d43bec014b662a69c51161d28b4822c322eea1fa7935fadb2fb52
SimHash 32204900c9ad

Groups

twitterbot

Rule Path
Disallow /mail.php
Disallow /search.php
Disallow /trackback/
Disallow /news/*.amp.html

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

*

Rule Path
Disallow /mail.php
Disallow /search.php
Disallow /trackback/
Back to top

Other Records

Field Value
sitemap https://www.golem.de/gsiteindex.xml
Back to top

Comments

  • Legal notice: golem.de expressly reserves the right to use its content for commercial text and data mining (ยง 44 b UrhG).
  • The use of robots or other automated means to access golem.de or collect or mine data without the express permission of golem.de is strictly prohibited.
  • golem.de may, in its discretion, permit certain automated access to certain golem.de pages.
  • If you would like to apply for permission to crawl golem.de, collect or use data, please email recht@golem.de
Back to top