Log In Sign Up

spiegel.de
robots.txt

Robots Exclusion Standard data for spiegel.de

Resource Scan

Scan Details

Site Domain spiegel.de
Base Domain spiegel.de
Scan Status Ok
Last Scan2024-04-23T22:35:47+00:00
Next Scan 2024-04-30T22:35:47+00:00

Last Scan

Scanned2024-04-23T22:35:47+00:00
URL https://spiegel.de/robots.txt
Redirect https://www.spiegel.de/robots.txt
Redirect Domain www.spiegel.de
Redirect Base spiegel.de
Domain IPs 128.65.210.8
Redirect IPs 128.65.210.180, 128.65.210.181, 128.65.210.182, 128.65.210.183, 128.65.210.184, 128.65.210.185
Response IP 128.65.210.181
Found Yes
Hash c2f22d76bc60113166a3e985e8065d4962e716308ff38ace1c436a12787dc1b0
SimHash b2240d5a6d81

Groups

*

Rule Path
Allow /
Disallow /*CR-Dokumentation.pdf$
Disallow /gutscheine/suche?
Disallow /gutscheine/*?code=*
Disallow /gutscheine/*%26code%3D*

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

sirdatabot

Rule Path
Disallow /

lcc

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.spiegel.de/sitemaps/news-de.xml
sitemap https://www.spiegel.de/sitemaps/videos/sitemap.xml
sitemap https://www.spiegel.de/plus/sitemap.xml
sitemap https://www.spiegel.de/sitemap.xml
sitemap https://www.spiegel.de/gutscheine/sitemap.xml
Back to top

Comments

  • Legal notice: spiegel.de expressly reserves the right to use its content for commercial text and data mining (ยง 44b Urheberrechtsgesetz).
  • The use of robots or other automated means to access spiegel.de or collect or mine data without the express permission of spiegel.de is strictly prohibited.
  • spiegel.de may, in its discretion, permit certain automated access to certain spiegel.de pages,
  • If you would like to apply for permission to crawl spiegel.de, collect or use data, please email syndication@spiegel.de
Back to top