Log In Sign Up

streetware.gr
robots.txt

Robots Exclusion Standard data for streetware.gr

Resource Scan

Scan Details

Site Domain streetware.gr
Base Domain streetware.gr
Scan Status Ok
Last Scan2025-12-19T11:16:31+00:00
Next Scan 2026-01-18T11:16:31+00:00

Last Scan

Scanned2025-12-19T11:16:31+00:00
URL https://streetware.gr/robots.txt
Redirect https://www.streetware.gr/robots.txt
Redirect Domain www.streetware.gr
Redirect Base streetware.gr
Domain IPs 195.201.104.236
Redirect IPs 195.201.104.236
Response IP 195.201.104.236
Found Yes
Hash d035c31532ac8017042b86600808c4659e9857b033384572ba7ec2e4c8f13a3b
SimHash 2444ad32eeb2

Groups

*

Rule Path
Allow /*?p=
Disallow */index.php/
Disallow */catalog/product_compare/
Disallow */catalog/category/view/
Disallow */catalog/product/view/
Disallow */catalog/product/gallery/
Disallow */catalogsearch/
Disallow */checkout/
Disallow */control/
Disallow */contacts/
Disallow */customer/
Disallow */customize/
Disallow */newsletter/
Disallow */poll/
Disallow */review/
Disallow */sales/
Disallow */sendfriend/
Disallow */tag/
Disallow */wishlist/
Disallow /*?dir*
Disallow /*?limit*
Disallow /*?mode*
Disallow /*?price=*
Disallow /*?___from_store=*
Disallow /*?___store=*
Disallow /*?q=*
Disallow /*?p=*&
Disallow /*.php$
Disallow /*?SID=
Back to top

Other Records

Field Value
sitemap https://www.streetware.gr/sitemap_el.xml
Back to top

Comments

  • For more information about the robots.txt standard visit
  • http://www.robotstxt.org/orig.html
  • For syntax checking visit
  • http://tool.motoricerca.info/robots-checker.phtml
  • Crawl the Sitemap. Set the correct URL before uncomment
  • Crawlers Setup
  • How many seconds a crawler should wait before loading and crawling page content
  • Set a custom crawl rate if you are experiencing traffic issues with your server
  • https://www.contentkingapp.com/academy/robotstxt/faq/crawl-delay-10/
  • Crawl-delay: 10
  • Allow to crawl paging (paging inside a listing with more params are disallowed below)
  • Do not crawl non-SEF paths and generated content (if you use a store id in URL you must prefix with * or copy for each store)
  • Allow: */catalogsearch/seo_sitemap
  • Allow: */catalogsearch/term/popular
  • Do not crawl dynamic filters. Uncomment what you need or add custom filters
  • Disallow: /*?cat=*
  • Disallow: /*?availability=*
  • Disallow: /*?brand=*
  • Do not crawl paths that can be safely ignored by search engines (no clean URLs)
  • Do not allow media indexing for the following bots
  • Disallow all or add custom paths. For example */media/ or */skin/
  • User-agent: baiduspider-image
  • Disallow: /
  • Disallow: */media/
  • Disallow: */skin/
  • User-agent: baiduspider-video
  • Disallow: /
  • Disallow: */media/
  • Disallow: */skin/
  • User-agent: msnbot-media
  • Disallow: /
  • Disallow: */media/
  • Disallow: */skin/
  • User-agent: Googlebot-Image
  • Disallow: /
  • Disallow: */media/
  • Disallow: */skin/
  • User-agent: Googlebot-Video
  • Disallow: /
  • Disallow: */media/
  • Disallow: */skin/
Back to top