Log In Sign Up

hs-esslingen.de
robots.txt

Robots Exclusion Standard data for hs-esslingen.de

Resource Scan

Scan Details

Site Domain hs-esslingen.de
Base Domain hs-esslingen.de
Scan Status Ok
Last Scan2024-05-08T19:59:41+00:00
Next Scan 2024-06-07T19:59:41+00:00

Last Scan

Scanned2024-05-08T19:59:41+00:00
URL https://hs-esslingen.de/robots.txt
Redirect https://www.hs-esslingen.de/robots.txt
Redirect Domain www.hs-esslingen.de
Redirect Base hs-esslingen.de
Domain IPs 134.108.34.3
Redirect IPs 134.108.34.3
Response IP 134.108.34.3
Found Yes
Hash e08cbdcc76a684a3ee3a295c2e74db5712dd7e7f856cab86435c9afadc5ea179
SimHash 3b9674ddc4dd

Groups

googlebot

Rule Path
Allow /

msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot/1.0 (+http://search.msn.com/msnbot.htm)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

rytebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

lrtbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

daumoa

Rule Path
Disallow /

yanga worldsearch bot

Rule Path
Disallow /

jobo

Rule Path
Disallow /

echo!

Rule Path
Disallow /

linkchecker

Rule Path
Disallow /

fish search

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

bloglines

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

appie

Rule Path
Disallow /

sun4u

Rule Path
Disallow /

httrack

Rule Path
Disallow /

nutch

Rule Path
Disallow /

sisi

Rule Path
Disallow /

fyberspider

Rule Path
Disallow /

girafabot

Rule Path
Disallow /

lmspider

Rule Path
Disallow /

np

Rule Path
Disallow /

robi

Rule Path
Disallow /

webster pro

Rule Path
Disallow /

webster

Rule Path
Disallow /

zeus

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

scirus

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

picosearch

Rule Path
Disallow /

plucker

Rule Path
Disallow /

disco pump

Rule Path
Disallow /

gulliver

Rule Path
Disallow /

vspider

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

teleport pro

Rule Path
Disallow /

fetch

Rule Path
Disallow /

pamuk

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webcapture

Rule Path
Disallow /

mass downloader

Rule Path
Disallow /

webcopy

Rule Path
Disallow /

awv0.8d

Rule Path
Disallow /

crescent internet toolpak

Rule Path
Disallow /

joc web spider

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

sitesucker

Rule Path
Disallow /

webdup

Rule Path
Disallow /

scooter

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

python

Rule Path
Disallow /

franklin locator

Rule Path
Disallow /

ck-sillydog

Rule Path
Disallow /

pockethttp

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

xaldon webspider

Rule Path
Disallow /

webcapture

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

java

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

teragramwebcrawler

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

nogoop-httpclient

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

myoperatb/1.0

Rule Path
Disallow /

myoperatb

Rule Path
Disallow /
Back to top

Comments

  • url: http://search.msn.com/msnbot.htm
  • note: msnbots have changed their name recently
  • url: http://www.msn.com/
  • note: msn also accepts a delay
  • url: http://search.msn.com/msnbot.htm
  • note: msnbots have changed their name recently
  • url: http://help.yahoo.com/help/us/ysearch/slurp/index.html
  • note: inktomi's "slurp" can read a minimum delay between hits
Back to top