Log In Sign Up

savilerowco.com
robots.txt

Robots Exclusion Standard data for savilerowco.com

Resource Scan

Scan Details

Site Domain savilerowco.com
Base Domain savilerowco.com
Scan Status Ok
Last Scan2024-05-23T14:39:37+00:00
Next Scan 2024-06-22T14:39:37+00:00

Last Scan

Scanned2024-05-23T14:39:37+00:00
URL https://savilerowco.com/robots.txt
Domain IPs 104.26.10.46, 104.26.11.46, 172.67.75.62, 2606:4700:20::681a:a2e, 2606:4700:20::681a:b2e, 2606:4700:20::ac43:4b3e
Response IP 104.26.10.46
Found Yes
Hash c751aaa4dc0f2abb8c859b31bccb610ec66e7f2a0d2211b77a95cfdaff61d926
SimHash ae02bb40c6a8

Groups

rogerbot

Rule Path
Disallow *

urlspy

Rule Path
Disallow /

*
mozilla/5.0 (compatible;nostocrawlerbot/1.0;+http://my.nosto.com/tagging)

Rule Path
Disallow /blackhole/
Disallow /sand/
Allow /*?p=
Allow /catalog/seo_sitemap/category/
Allow /catalog/seo_sitemap/product/
Allow /media/catalog/product
Disallow /404/
Disallow /api/
Disallow /app/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /install/
Disallow /js/
Disallow /lib/
Disallow /magento/
Disallow /pkginfo/
Disallow /private/
Disallow /report/
Disallow /scripts/
Disallow /shell/
Disallow /skin/
Disallow /stats/
Disallow /upload/
Disallow /var/
Disallow /6373bac9c22a8/
Disallow /index.php/
Disallow /catalog/product/view/
Disallow /catalog/category/view/
Disallow /catalog/product_compare/
Disallow /checkout/
Disallow /checkout/cart/
Disallow /checkout/cart/add/
Disallow /control/
Disallow /contacts/index/
Disallow /contacts/index/post/
Disallow /customer/
Disallow /customize/
Disallow /newsletter/
Disallow /poll/
Disallow /review/
Disallow /sendfriend/
Disallow /tag/
Disallow /wishlist/
Disallow /made-to-measure/personalise/
Disallow /made-to-measure/measure/
Disallow /made-to-measure-category/
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /STATUS.txt
Allow /*.js$
Allow /*.css$
Disallow /*.php$
Disallow /*?p=*&
Disallow /*?SID=
Disallow /enable-cookies
Disallow /service-unavailable
Disallow /helpful-support
Disallow /ocado_offer
Disallow /offer
Disallow /test
Disallow /staging/stg_main/
Disallow /new-customers/
Disallow /short
Disallow /lookbook
Disallow */size-
Disallow *chest_size
Disallow *colour_
Disallow *--
Disallow *fabric_type
Disallow *clearance/design-
Disallow *shirts/design-
Disallow *collar_type
Disallow *sleeve_length
Disallow *cuff_type
Disallow *fit-slim
Disallow *fit-classic
Disallow *fit-tailored
Disallow *jacket_length
Disallow *waist_size
Disallow *leg_length
Disallow *color_group
Disallow *cat-clearance
Disallow *promo_flag
Disallow *cat-casual_wear
Disallow *cat-shirts
Disallow *cat-suits
Disallow *cat-accessories
Disallow *filtercategoryROOT
Disallow *filterdesign
Back to top

Other Records

Field Value
sitemap https://savilerowco.com/sitemap_index.xml
Back to top

Comments

  • XML Sitemap
  • Begin UA Bans
  • Block RogerBot - added by Jan on the 15th April 2019
  • End UA Bans
  • Crawlers Setup
  • Allow all to Nosto crawler
  • Blackholes to kill those who fail to honour robots.txt
  • and Sand Traps for email addresses scrapers
  • Allowable Index
  • Disallow: /catalogsearch/result/
  • Allowable images so you will show in image search and shopping networks
  • Directories
  • Paths (clean URLs)
  • Disallow: /catalogsearch/
  • Disallow: /catalogsearch/advanced/
  • Disallow: /catalogsearch/advanced/result/
  • Disallow: /catalogsearch/term/
  • Disallow: /catalogsearch/term/popular/
  • Disallow: /contacts/
  • Disallow: /customer/account/
  • Disallow: /customer/account/login/
  • Added by Jan, MTM step links, on 12th Nov 2018
  • Added by DT, MTM non-canonical, on 9th Sept 2022
  • Files
  • Paths (no clean URLs)
  • Allowed by Jan 14.9.2018, recommended by new SEO agency
  • Filter Block
  • Block old FactFinder filter links, added by Jan 19.8.2021
Back to top

Warnings

  • 2 invalid lines.
Back to top