Log In Sign Up

werder.de
robots.txt

Robots Exclusion Standard data for werder.de

Resource Scan

Scan Details

Site Domain werder.de
Base Domain werder.de
Scan Status Ok
Last Scan2024-06-24T03:02:57+00:00
Next Scan 2024-07-24T03:02:57+00:00

Last Scan

Scanned2024-06-24T03:02:57+00:00
URL https://werder.de/robots.txt
Redirect https://www.werder.de/robots.txt
Redirect Domain www.werder.de
Redirect Base werder.de
Domain IPs 185.48.220.200
Redirect IPs 185.48.220.200
Response IP 185.48.220.200
Found Yes
Hash bad0ef543c715fdd15549420475e034075d6b9922988635fdaad80e25c489914
SimHash 986153608ebb

Groups

*

Rule Path
Disallow /forum/?*
Disallow /forum/attachments/
Disallow /forum/forums/*/?direction
Disallow /forum/forums/*/?order
Disallow /forum/help/
Disallow /forum/login/
Disallow /forum/lost-password/
Disallow /forum/members/
Disallow /forum/misc/
Disallow /forum/online/
Disallow /forum/posts/
Disallow /forum/register/
Disallow /forum/search/
Disallow /forum/goto/
Disallow /*.git$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$
Disallow /*.tar.gz$
Disallow /*.zip$
Disallow /session.php
Disallow /shop/admin/
Disallow /shop/app/
Disallow /shop/downloader/
Disallow /shop/errors/
Disallow /shop/includes/
Disallow /shop/lib/
Disallow /shop/pkginfo/
Disallow /shop/shell/
Disallow /shop/skin/
Disallow /shop/var/
Disallow /shop/zu-hause/sky.html
Disallow /shop/sizes-table
Disallow /shop/api.php
Disallow /shop/cron.php
Disallow /shop/cron.sh
Disallow /shop/error_log
Disallow /shop/get.php
Disallow /shop/install.php
Disallow /shop/LICENSE.html
Disallow /shop/LICENSE.txt
Disallow /shop/LICENSE_AFL.txt
Disallow /shop/README.txt
Disallow /shop/RELEASE_NOTES.txt
Disallow /shop/STATUS.txt
Disallow /shop/*?Dir*
Disallow /shop/*?Dir=desc
Disallow /shop/*?Dir=asc
Disallow /shop/*?Limit=all
Disallow /shop/*?Mode*
Disallow /shop/index.php/
Disallow /shop/*?SID=
Disallow /*?*esi=1*
Disallow /*/esi/*
Disallow /shop/catalog/product_compare/
Disallow /shop/checkout/
Disallow /shop/onestepcheckout/
Disallow /shop/newsletter/
Disallow /shop/poll/
Disallow /shop/review/
Disallow /shop/sendfriend/
Disallow /shop/en/catalog/product_compare/
Disallow /shop/en/checkout/
Disallow /shop/en/onestepcheckout/
Disallow /shop/en/newsletter/
Disallow /shop/en/poll/
Disallow /shop/en/review/
Disallow /shop/en/sendfriend/
Disallow /shop/catalog/category/view/
Disallow /shop/catalog/product/view/
Disallow /shop/catalog/product/gallery/
Disallow /shop/en/catalog/category/view/
Disallow /shop/en/catalog/product/view/
Disallow /shop/en/catalog/product/gallery/

mj12bot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

fatbot

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

trendictionbot0.5.0

Rule Path
Disallow /

owlin

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

stashbot

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

naver

Rule Path
Disallow /

yeti

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

squidbot

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

veooz

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.werder.de/shop/media/sitemap.xml
sitemap https://www.werder.de/shop/media/sitemap_en.xml
Back to top

Comments

  • Enables robots.txt rules for all crawlers
  • Disallow forum paths
  • Do not allow indexing files and folders that are required during development: CVS, SVN directory and dump files
  • GENERAL SETTINGS For MAGENTO
  • Do not index the general technical Magento directory
  • Do not index the shared files Magento
  • MAGENTO SEA IMPROVEMENT
  • Do not index the page subcategories that are sorted or filtered.
  • Do not index the second copy of the home page (example.com / index.php /). Un-comment only if you have activated Magento SEO URLs.
  • Do not index the link from the session ID
  • disallow all esi parameters
  • Do not index the page checkout and user account
  • Do not index the search page and CEO, non-optimized link categories
  • Exclude Various Robots
Back to top