Log In Sign Up

bau-welt.de
robots.txt

Robots Exclusion Standard data for bau-welt.de

Resource Scan

Scan Details

Site Domain bau-welt.de
Base Domain bau-welt.de
Scan Status Ok
Last Scan2024-09-29T04:06:48+00:00
Next Scan 2024-10-06T04:06:48+00:00

Last Scan

Scanned2024-09-29T04:06:48+00:00
URL https://bau-welt.de/robots.txt
Redirect https://www.bau-welt.de/robots.txt
Redirect Domain www.bau-welt.de
Redirect Base bau-welt.de
Domain IPs 195.122.144.60
Redirect IPs 195.122.144.60
Response IP 195.122.144.60
Found Yes
Hash 4f89ada7c3b2407d27a5cc8a7e238f42f157524bf525e49ee411e842b015c89f
SimHash 3311135ffef0

Groups

*

Rule Path
Allow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path Comment
Disallow / -
Disallow /prototype/ -
Disallow /typo3/ -
Disallow /typo3conf/ -
Allow /typo3conf/ext/ -
Allow /typo3temp/ -
Disallow /*?id=* no non-speaking URLs
Disallow /*%26id%3D* no non-speaking URLs
Disallow /*?L=0* no default lang
Disallow /*%26L%3D0* no default lang
Disallow /*cHash no cHash
Disallow /*?type=98 no print pages
Disallow /*%26type%3D98 no print pages
Disallow /*tx_form_formframework no forms
Disallow /*tx_solr%5Bq%5D%3D* no search results
Disallow /*tx_solr*%3Dkategorie* forbidden filter
Disallow /*tx_solr*%3Dwohnflaeche* forbidden filter
Disallow /*tx_solr*%3Dpreis* forbidden filter
Disallow /*tx_solr* no solr results of any kind
Allow /*sitemap%3D*%26cHash%3D* -
Back to top

Other Records

Field Value
sitemap https://www.bau-welt.de/sitemap.xml
Back to top

Comments

  • disallow ai crawler
  • allow assets and assets of extensions
  • parameters (keep the index clean)
  • sitemap
  • allow sitemaps with chash
  • robots.txt for site bau-welt.de
  • NB: file is cached (ttl 4h)
Back to top