Log In Sign Up

onlinestreet.de
robots.txt

Robots Exclusion Standard data for onlinestreet.de

Resource Scan

Scan Details

Site Domain onlinestreet.de
Base Domain onlinestreet.de
Scan Status Ok
Last Scan2024-09-30T02:42:34+00:00
Next Scan 2024-10-07T02:42:34+00:00

Last Scan

Scanned2024-09-30T02:42:34+00:00
URL https://onlinestreet.de/robots.txt
Domain IPs 85.25.185.102
Response IP 85.25.185.102
Found Yes
Hash fe26075af3304d1bf7e7d7034dab7ba574ad7067034a56723170a4b0ea77d393
SimHash a01652b0c3b0

Groups

*

Rule Path Comment
Disallow /strassen/schild/ DISALLOW: Strassenschilder
Allow /strassen/schild/generator.html ALLOW: STRASSENSCHILD GENERATOR
Disallow /strassen/ortsschild/*?download=1 Ortsschilder: Download
Disallow /suche/ Suche

dotbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3
Back to top

Comments

  • DISALLOW
  • TOO HUNGRY
  • https://moz.com/researchtools/ose/dotbot
  • https://dataforseo.com/dataforseo-bot
  • https://help.ahrefs.com/en/articles/78158-how-do-i-control-your-bot-s-crawling-behaviour
  • https://www.trendiction.com/bot
  • https://megaindex.com/crawler
  • https://www.semrush.com/bot/
  • https://www.feitsui.com/en/article/32
  • low priority for bing
  • low priority for Yahoo
Back to top