Log In Sign Up

db-syltshuttle.de
robots.txt

Robots Exclusion Standard data for db-syltshuttle.de

Resource Scan

Scan Details

Site Domain db-syltshuttle.de
Base Domain db-syltshuttle.de
Scan Status Ok
Last Scan2025-10-03T05:41:16+00:00
Next Scan 2025-10-17T05:41:16+00:00

Last Scan

Scanned2025-10-03T05:41:16+00:00
URL https://db-syltshuttle.de/robots.txt
Redirect https://www.frs-syltfaehre.de/robots.txt
Redirect Domain www.frs-syltfaehre.de
Redirect Base frs-syltfaehre.de
Domain IPs 104.21.15.80, 172.67.161.245, 2606:4700:3032::6815:f50, 2606:4700:3036::ac43:a1f5
Redirect IPs 172.66.41.46, 172.66.42.210, 2606:4700:3108::ac42:292e, 2606:4700:3108::ac42:2ad2
Response IP 172.66.42.210
Found Yes
Hash 140b23f57343d3f4ce3cf7e6bde3a1a539532bccef6ff8cc4df0d284cdaed934
SimHash 64d9fb476941

Groups

*

Rule Path
Disallow /*?id=*
Disallow /*%26id%3D*
Disallow /*?quot
Disallow /*?wbraid
Disallow /*?gbraid
Disallow /*?tag=*
Disallow /*?L=0*
Disallow /*%26L%3D0*
Disallow /*/Private/*
Disallow /*/Configuration/*
Disallow /typo3temp/*
Allow /typo3temp/*.css$
Allow /typo3temp/*.css.*.gzip$
Allow /typo3temp/*.js$
Allow /typo3temp/*.js.*.gzip$
Allow /typo3temp/*.jpg$
Allow /typo3temp/*.gif$
Allow /typo3temp/*.png$
Allow /typo3temp/*.pdf$
Disallow *.sql
Disallow *.sql.gz
Disallow /*.pdf$
Back to top

Comments

  • Specialized robots.txt for TYPO3
  • See: https://gist.github.com/oliverthiele/83957820413fd981e062
  • Only allow URLs generated with RealURL
  • Disallow crawling of URLs with UTM parameters
  • L=0 is the default language
  • Should always be protected (.htaccess)
  • Disallow temporary files
  • Disallow SQL files
  • Disallow all PDF from being indexed
  • Temporarily Disallow links
Back to top

Warnings

  • 12 invalid lines.
Back to top