Log In Sign Up

uni-essen.de
robots.txt

Robots Exclusion Standard data for uni-essen.de

Resource Scan

Scan Details

Site Domain uni-essen.de
Base Domain uni-essen.de
Scan Status Ok
Last Scan2025-07-28T14:14:19+00:00
Next Scan 2025-08-27T14:14:19+00:00

Last Scan

Scanned2025-07-28T14:14:19+00:00
URL http://www.uni-essen.de/robots.txt
Redirect https://www.uni-due.de/robots.txt
Redirect Domain www.uni-due.de
Redirect Base uni-due.de
Domain IPs 134.91.197.148
Redirect IPs 134.91.197.147
Response IP 134.91.197.147
Found Yes
Hash cc045139be2768522de42d246cb3b4028e75654e13479c3017238b33c1f5ea57
SimHash 67d4f844a3b2

Groups

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

gptbot

Rule Path
Disallow /
Back to top

Comments

  • robots.txt file
  • Bans all robots from listed directories
  • User-agent: *
  • Disallow: /~ht0209/lab2014/
  • Disallow: /~bi0058/
  • Disallow: /cms/zs-2001/htdocs/
  • Disallow: /imp/zs-2001/htdocs/
  • Disallow: /~ht0209/lab2014/
  • Disallow: /~bi0058/
Back to top