Log In Sign Up

shuz.nl
robots.txt

Robots Exclusion Standard data for shuz.nl

Resource Scan

Scan Details

Site Domain shuz.nl
Base Domain shuz.nl
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-07-03T07:51:59+00:00
Next Scan 2025-10-01T07:51:59+00:00

Last Successful Scan

Scanned2024-05-17T05:20:14+00:00
URL https://shuz.nl/robots.txt
Redirect https://www.shuz.nl/robots.txt
Redirect Domain www.shuz.nl
Redirect Base shuz.nl
Domain IPs 162.55.243.227
Response IP 162.55.243.227
Found Yes
Hash 6c336da130aef7a36aa5aef99c89fcaba2556492493600a7832e212ef18afe4f
SimHash bd20f97f83a0

Groups

*

Rule Path
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /setup/
Disallow /update/
Disallow /var/
Disallow /vendor/
Disallow /*?tn_cid
Disallow /*?tn_fk

semrush-bot

Rule Path
Disallow

*

Rule Path
Disallow /index.php/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /catalogsearch/
Disallow /control/
Disallow /contacts/
Disallow /customer/
Disallow /customize/
Disallow /newsletter/
Disallow /wishlist/
Disallow /customer/account/
Disallow /customer/account/login/

*

Rule Path
Disallow /composer.json
Disallow /composer.lock
Disallow /CONTRIBUTING.md
Disallow /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow /COPYING.txt
Disallow /Gruntfile.js
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /nginx.conf.sample
Disallow /package.json
Disallow /php.ini.sample
Disallow /RELEASE_NOTES.txt
Disallow /*?Dir*
Disallow /*?Dir=desc
Disallow /*?Dir=asc
Disallow /*?Limit=all
Disallow /*?Mode*
Disallow /*.php$
Disallow /*?SID=
Disallow /*.CVS
Disallow /*.Zip$
Disallow /*.Svn$
Disallow /*.Idea$
Disallow /*.Sql$
Disallow /*.Tgz$

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://www.shuz.nl/media/siteindex.xml
Back to top

Comments

  • Website Sitemap
  • Crawlers Setup
  • Directories
  • Disallow: /pub/
  • Fix googlebot
  • Semrush bot
  • Paths (clean URLs)
  • Do not index the general technical directories and files on a server
  • Disallow: /cgi-bin/
  • Files
  • Do not index the page subcategories that are sorted or filtered.
  • Do not index the link from the session ID
  • CVS, SVN directory and dump files
Back to top

Warnings

  • `noindex` is not a known field.
Back to top