Log In Sign Up

zeit.de
robots.txt

Robots Exclusion Standard data for zeit.de

Resource Scan

Scan Details

Site Domain zeit.de
Base Domain zeit.de
Scan Status Ok
Last Scan2024-05-02T19:24:29+00:00
Next Scan 2024-05-09T19:24:29+00:00

Last Scan

Scanned2024-05-02T19:24:29+00:00
URL https://zeit.de/robots.txt
Redirect https://www.zeit.de/robots.txt
Redirect Domain www.zeit.de
Redirect Base zeit.de
Domain IPs 35.246.185.66
Redirect IPs 151.101.130.49, 151.101.194.49, 151.101.2.49, 151.101.66.49
Response IP 199.232.46.49
Found Yes
Hash b206c99539d2a1f8886300fa5c131a7422143022cce8508100edce0af5c24c7d
SimHash 2224445c4d8f

Groups

googlebot-news

Rule Path
Disallow /angebote/

*

Rule Path
Disallow /zeit/
Disallow /templates/
Disallow /hp_channels/
Disallow /send/
Disallow /suche/
Disallow /rezepte/suche/
Disallow */comment-thread?
Disallow */liveblog-backend*
Disallow /framebuilder/
Disallow /campus/framebuilder/
Disallow /cre-1.0/tracking/*.js$
Disallow /navigation-teasers*

baiduspider

Rule Path
Disallow /

applebot

Rule Path
Allow /
Disallow /cre-1.0/

grapeshotcrawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.zeit.de/gsitemaps/index.xml
Back to top

Comments

  • Legal notice: zeit.de expressly reserves the right to use its content for commercial text and data mining (ยง 44 b UrhG).
  • The use of robots or other automated means to access zeit.de or collect or mine data without
  • the express permission of zeit.de is strictly prohibited.
  • zeit.de may, in its discretion, permit certain automated access to certain zeit.de pages,
  • If you would like to apply for permission to crawl zeit.de, collect or use data, please email online-syndication@zeit.de
Back to top