Log In Sign Up

frs.es
robots.txt

Robots Exclusion Standard data for frs.es

Resource Scan

Scan Details

Site Domain frs.es
Base Domain frs.es
Scan Status Ok
Last Scan2025-04-17T02:44:36+00:00
Next Scan 2025-05-17T02:44:36+00:00

Last Scan

Scanned2025-04-17T02:44:36+00:00
URL https://frs.es/robots.txt
Redirect https://www.frs.es/robots.txt
Redirect Domain www.frs.es
Redirect Base frs.es
Domain IPs 172.66.41.30, 172.66.42.226, 2606:4700:3108::ac42:291e, 2606:4700:3108::ac42:2ae2
Redirect IPs 172.66.41.30, 172.66.42.226, 2606:4700:3108::ac42:291e, 2606:4700:3108::ac42:2ae2
Response IP 172.66.41.30
Found Yes
Hash 140b23f57343d3f4ce3cf7e6bde3a1a539532bccef6ff8cc4df0d284cdaed934
SimHash 64d9fb476941

Groups

*

Rule Path
Disallow /*?id=*
Disallow /*%26id%3D*
Disallow /*?quot
Disallow /*?wbraid
Disallow /*?gbraid
Disallow /*?tag=*
Disallow /*?L=0*
Disallow /*%26L%3D0*
Disallow /*/Private/*
Disallow /*/Configuration/*
Disallow /typo3temp/*
Allow /typo3temp/*.css$
Allow /typo3temp/*.css.*.gzip$
Allow /typo3temp/*.js$
Allow /typo3temp/*.js.*.gzip$
Allow /typo3temp/*.jpg$
Allow /typo3temp/*.gif$
Allow /typo3temp/*.png$
Allow /typo3temp/*.pdf$
Disallow *.sql
Disallow *.sql.gz
Disallow /*.pdf$
Back to top

Comments

  • Specialized robots.txt for TYPO3
  • See: https://gist.github.com/oliverthiele/83957820413fd981e062
  • Only allow URLs generated with RealURL
  • Disallow crawling of URLs with UTM parameters
  • L=0 is the default language
  • Should always be protected (.htaccess)
  • Disallow temporary files
  • Disallow SQL files
  • Disallow all PDF from being indexed
  • Temporarily Disallow links
Back to top

Warnings

  • 12 invalid lines.
Back to top