Log In Sign Up

uni-hohenheim.de
robots.txt

Robots Exclusion Standard data for uni-hohenheim.de

Resource Scan

Scan Details

Site Domain uni-hohenheim.de
Base Domain uni-hohenheim.de
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-09-29T08:23:05+00:00
Next Scan 2024-10-06T08:23:05+00:00

Last Successful Scan

Scanned2024-08-29T08:22:16+00:00
URL https://uni-hohenheim.de/robots.txt
Redirect https://www.uni-hohenheim.de/robots.txt
Redirect Domain www.uni-hohenheim.de
Redirect Base uni-hohenheim.de
Domain IPs 144.41.103.105
Redirect IPs 144.41.103.105, 2001:7c0:2700:1103::105
Response IP 144.41.103.105
Found Yes
Hash bab866776abdf5b4626bdade23fe758db77d49633da9a780f2205fddf5fea849
SimHash b89e55ddc4dc

Groups

*

Rule Path
Disallow /en/search?q=
Disallow /suche?q=

*

Rule Path
Disallow /spraz/
Disallow /*?state=wplan
Disallow /*%26state%3Dwplan
Disallow /en/course-catalogue?
Disallow /vorlesungs-verzeichnis?
Disallow /86260?
Disallow /index.php?id=86260&state=wplan
Disallow /118127?
Disallow /index.php?id=118127&state=wplan
Disallow /en/118127?state=wplan
Disallow /118129?
Disallow /en/timetables?
Disallow /index.php?id=1568

msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot/1.0 (+http://search.msn.com/msnbot.htm)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mozilla/5.0 (compatible; msie 10.0; windows nt 6.1; trident/6.0) response by siteimprove.com

Rule Path
Disallow /

buck

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

drupact

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

seznambot
semrushbot

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

tracemyfile

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

scrapy

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

spbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

sqlmap

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

yanga worldsearch bot

Rule Path
Disallow /

jobo

Rule Path
Disallow /

echo!

Rule Path
Disallow /

linkchecker

Rule Path
Disallow /

fish search

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

bloglines

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

appie

Rule Path
Disallow /

sun4u

Rule Path
Disallow /

httrack

Rule Path
Disallow /

nutch

Rule Path
Disallow /

sisi

Rule Path
Disallow /

fyberspider

Rule Path
Disallow /

girafabot

Rule Path
Disallow /

lmspider

Rule Path
Disallow /

np

Rule Path
Disallow /

robi

Rule Path
Disallow /

webster pro

Rule Path
Disallow /

webster

Rule Path
Disallow /

zeus

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

scirus

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

picosearch

Rule Path
Disallow /

plucker

Rule Path
Disallow /

disco pump

Rule Path
Disallow /

gulliver

Rule Path
Disallow /

vspider

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

teleport pro

Rule Path
Disallow /

fetch

Rule Path
Disallow /

pamuk

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webcapture

Rule Path
Disallow /

mass downloader

Rule Path
Disallow /

webcopy

Rule Path
Disallow /

awv0.8d

Rule Path
Disallow /

crescent internet toolpak

Rule Path
Disallow /

joc web spider

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

sitesucker

Rule Path
Disallow /

webdup

Rule Path
Disallow /

scooter

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

python

Rule Path
Disallow /

franklin locator

Rule Path
Disallow /

ck-sillydog

Rule Path
Disallow /

pockethttp

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

xaldon webspider

Rule Path
Disallow /

webcapture

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

java

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

teragramwebcrawler

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

nogoop-httpclient

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

myoperatb/1.0

Rule Path
Disallow /

myoperatb

Rule Path
Disallow /

yandex

Rule Path
Disallow /

vebidoobot

Rule Path
Disallow /

newspaper

Rule Path
Disallow /
Back to top

Comments

  • url: http://search.msn.com/msnbot.htm
  • note: msnbots have changed their name recently
  • url: http://www.msn.com/
  • note: msn also accepts a delay
  • url: http://search.msn.com/msnbot.htm
  • note: msnbots have changed their name recently
  • url: http://help.yahoo.com/help/us/ysearch/slurp/index.html
  • note: inktomi's "slurp" can read a minimum delay between hits
Back to top

Warnings

  • `request-rate` is not a known field.
Back to top