Log In Sign Up

surfundski.de
robots.txt

Robots Exclusion Standard data for surfundski.de

Resource Scan

Scan Details

Site Domain surfundski.de
Base Domain surfundski.de
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-11-23T01:48:49+00:00
Next Scan 2026-01-22T01:48:49+00:00

Last Successful Scan

Scanned2025-09-01T10:23:31+00:00
URL https://surfundski.de/robots.txt
Domain IPs 104.21.17.155, 172.67.177.15, 2606:4700:3033::ac43:b10f, 2606:4700:3036::6815:119b
Response IP 172.67.177.15
Found Yes
Hash fcf63b8e0bf87430084511af613a96ba1b4ea7174a81d223acb96f220c97c832
SimHash dd0c5cc6c184

Groups

*

Rule Path
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /var/
Disallow /report/
Disallow /pkginfo/
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /sendfriend/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /review/
Disallow /tag/
Disallow /index.php/
Disallow /*.php$
Disallow /*SID%3D
Disallow /composer.json
Disallow /composer.lock
Disallow /CONTRIBUTING.md
Disallow /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow /COPYING.txt
Disallow /Gruntfile.js
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /nginx.conf.sample
Disallow /package.json
Disallow /php.ini.sample
Disallow /RELEASE_NOTES.txt
Disallow /*.git
Disallow /*.CVS
Disallow /*.Zip$
Disallow /*.Svn$
Disallow /*.Idea$
Disallow /*.Sql$
Disallow /*.Tgz$
Disallow /*?*product_list_mode=
Disallow /*?*product_list_order=
Disallow /*?*product_list_limit=
Disallow /*?*product_list_dir=
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /*?gender=*
Disallow /*?size=*
Disallow /*?butikslager=*
Disallow /*?Farve=*
Disallow /*?mgs_brand=*
Disallow /*?zegul_model=*
Disallow /*?materiale=*
Disallow /*?*gender=*&*size=*
Disallow /*?*gender=*&*butikslager=*
Disallow /*?*gender=*&*Farve=*
Disallow /*?*gender=*&*mgs_brand=*
Disallow /*?*gender=*&*zegul_model=*
Disallow /*?*gender=*&*materiale=*
Disallow /*?*size=*&*butikslager=*
Disallow /*?*size=*&*Farve=*
Disallow /*?*size=*&*mgs_brand=*
Disallow /*?*size=*&*zegul_model=*
Disallow /*?*size=*&*materiale=*
Disallow /*?*butikslager=*&*Farve=*
Disallow /*?*butikslager=*&*mgs_brand=*
Disallow /*?*butikslager=*&*zegul_model=*
Disallow /*?*butikslager=*&*materiale=*
Disallow /*?*Farve=*&*mgs_brand=*
Disallow /*?*Farve=*&*zegul_model=*
Disallow /*?*Farve=*&*materiale=*
Disallow /*?*mgs_brand=*&*zegul_model=*
Disallow /*?*mgs_brand=*&*materiale=*
Disallow /*?*zegul_model=*&*materiale=*
Allow /*.css
Allow /*.js

Other Records

Field Value
crawl-delay 20

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /
Back to top

Comments

  • Magento sensitive or non-indexable paths
  • System/config files
  • Version control / backups
  • Magento layered navigation, sort, limit
  • Block individual filter attributes
  • Block all combinations of these attributes (regardless of order)
  • You can optionally add 3+ param combinations if needed
  • Allow essential CSS and JS
  • Block known bad bots
Back to top