Log In Sign Up

uni-bremen.de
robots.txt

Robots Exclusion Standard data for uni-bremen.de

Resource Scan

Scan Details

Site Domain uni-bremen.de
Base Domain uni-bremen.de
Scan Status Ok
Last Scan2024-09-09T13:16:12+00:00
Next Scan 2024-10-09T13:16:12+00:00

Last Scan

Scanned2024-09-09T13:16:12+00:00
URL https://uni-bremen.de/robots.txt
Redirect https://www.uni-bremen.de/robots.txt
Redirect Domain www.uni-bremen.de
Redirect Base uni-bremen.de
Domain IPs 134.102.22.124, 2001:638:708:16::22:124
Redirect IPs 134.102.22.124, 2001:638:708:16::22:124
Response IP 134.102.22.124
Found Yes
Hash d4dc9934c567fa4b6a98c93e7f1d5261139c6244024e1690b6d22fa677aa6014
SimHash 67991b554b80

Groups

*

Rule Path
Disallow /*?id=*
Disallow /*%26id%3D*
Disallow /*?L=0*
Disallow /*%26L%3D0*
Disallow /suchen
Disallow /search
Disallow /suchen?q
Disallow /search?q
Disallow /*?tx_solr%5Bq%5D=
Disallow /*?type=98*
Disallow /*%26type%3D98*
Disallow /*/Private/*
Disallow /fileadmin/templates/html/*
Disallow /*/Configuration/*
Disallow /typo3temp/*
Disallow *.sql
Disallow *.sql.gz

googlebot

Rule Path
Allow /typo3temp/*.css$
Allow /typo3temp/*.css.*.gzip$
Allow /typo3temp/*.js$
Allow /typo3temp/*.js.*.gzip$
Allow /typo3temp/*.jpg$
Allow /typo3temp/*.gif$
Allow /typo3temp/*.png$

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.uni-bremen.de/sitemap.xml
Back to top

Comments

  • Only allow URLs generated with RealURL
  • L=0 is the default language
  • no search results
  • typeNum = 98 is usually the print version.
  • Should always be protected (.htaccess)
  • Crawl - delay
Back to top