Log In Sign Up

htw-saarland.de
robots.txt

Robots Exclusion Standard data for htw-saarland.de

Resource Scan

Scan Details

Site Domain htw-saarland.de
Base Domain htw-saarland.de
Scan Status Ok
Last Scan2025-07-16T09:24:57+00:00
Next Scan 2025-08-15T09:24:57+00:00

Last Scan

Scanned2025-07-16T09:24:57+00:00
URL http://htw-saarland.de/robots.txt
Redirect https://www.htwsaar.de/robots.txt
Redirect Domain www.htwsaar.de
Redirect Base htwsaar.de
Domain IPs 134.96.210.180
Redirect IPs 134.96.210.180
Response IP 134.96.210.180
Found Yes
Hash 0dabb521d2c6e1a6e4ec35253cfc88116693f1bc6734a77670e736795aaba139
SimHash ac71ab554d61

Groups

*

Rule Path
Disallow /

googlebot

Rule Path
Disallow /*?
Disallow /*atct_album_view$
Disallow /*folder_factories$
Disallow /*folder_summary_view$
Disallow /*login_form$
Disallow /*mail_password_form$
Disallow /%40%40search
Disallow /%40%40searchView
Disallow /*search_rss$
Disallow /*sendto_form$
Disallow /*summary_view$
Disallow /*thumbnail_view$
Disallow /*view$
Back to top

Other Records

Field Value
sitemap https://www.htwsaar.de/sitemap.xml.gz
Back to top

Comments

  • Define access-restrictions for robots/spiders
  • http://www.robotstxt.org/wc/norobots.html
  • By default we allow robots to access all areas of our site
  • already accessible to anonymous users
  • Add Googlebot-specific syntax extension to exclude forms
  • that are repeated for each piece of content in the site
  • the wildcard is only supported by Googlebot
  • http://www.google.com/support/webmasters/bin/answer.py?answer=40367&ctx=sibling
Back to top