Log In Sign Up

muehlacker-tagblatt.de
robots.txt

Robots Exclusion Standard data for muehlacker-tagblatt.de

Resource Scan

Scan Details

Site Domain muehlacker-tagblatt.de
Base Domain muehlacker-tagblatt.de
Scan Status Ok
Last Scan2024-06-28T11:47:41+00:00
Next Scan 2024-07-05T11:47:41+00:00

Last Scan

Scanned2024-06-28T11:47:41+00:00
URL https://muehlacker-tagblatt.de/robots.txt
Redirect https://www.muehlacker-tagblatt.de/robots.txt
Redirect Domain www.muehlacker-tagblatt.de
Redirect Base muehlacker-tagblatt.de
Domain IPs 217.182.187.120
Redirect IPs 217.182.187.120
Response IP 217.182.187.120
Found Yes
Hash f5b0d4f424bc20e4341fe88b46674ee379c2e293c19398e73dd6532a5f7c098f
SimHash b3205f10cda7

Groups

*

Rule Path
Disallow /User
Disallow /Dateien
Disallow /Nachrichten/Suche
Disallow /ScriptResource
Disallow /WebResource

meltwater

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

bloodhound

Rule Path
Disallow /

cydralspider

Rule Path
Disallow /

downloadexpress

Rule Path
Disallow /

gammaspider

Rule Path
Disallow /

objectssearch

Rule Path
Disallow /

pimptrain

Rule Path
Disallow /

raven

Rule Path
Disallow /

wapspider

Rule Path
Disallow /

webzinger

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 2
Back to top

Comments

  • Robots.txt for crawler
  • Disallow Crawler
  • Crawler often creates invalid script/webresource resource request
  • Max crawler Time per page in sec
  • Sitemap
  • Sitemap: https://www.muehlacker-tagblatt.de/Sitemap_Index.xml.gz
  • Legal notice: muehlacker-tagblatt.de / Karl Elser GmbH expressly reserves the right to use its content for commercial text and data mining (� 44 b UrhG).
  • The use of robots or other automated means to access muehlacker-tagblatt.de or collect or mine data without the express permission of muehlacker-tagblatt.de / Karl Elser GmbH is strictly prohibited.
  • muehlacker-tagblatt.de / Karl Elser GmbH may, in its discretion, permit certain automated access to certain muehlacker-tagblatt.de pages.
  • If you would like to apply for permission to crawl muehlacker-tagblatt.de / Karl Elser GmbH, collect or use data, please email info@muehlacker-tagblatt.de
Back to top