Log In Sign Up

motolegends.com
robots.txt

Robots Exclusion Standard data for motolegends.com

Resource Scan

Scan Details

Site Domain motolegends.com
Base Domain motolegends.com
Scan Status Ok
Last Scan2024-09-09T13:46:58+00:00
Next Scan 2024-10-09T13:46:58+00:00

Last Scan

Scanned2024-09-09T13:46:58+00:00
URL https://motolegends.com/robots.txt
Redirect https://www.motolegends.com/robots.txt
Redirect Domain www.motolegends.com
Redirect Base motolegends.com
Domain IPs 93.184.249.29
Redirect IPs 93.184.249.29
Response IP 93.184.249.29
Found Yes
Hash 78495eec3c192ff7526b41ee8b20d19a17447692dacd5cc39a1e33f4390403ab
SimHash 7617f17049b6

Groups

*

Rule Path
Disallow /*?
Disallow /login
Disallow /basket*
Disallow /Basket.aspx*
Disallow /checkout
Disallow /*?*basket_action=1*
Disallow /*?*interaction=*
Disallow /*?************************************************************************************donotindex=1*
Disallow /*?***************************************************************filter=*
Disallow /*?***************************************************************fnsetsortorder=*
Disallow /*****************************************************************?*sortorder=*
Disallow /*****************************************************************?*listing_pagesize=*
Disallow /*****************************************************************?*fnsetpagesize=*
Disallow /jackets?*Product-Type=Jackets
Disallow /jackets?*Product-Type=Jacket
Disallow /helmets?*Product-Type=Helmets
Disallow /boots?*Product-Type=Boots
Disallow /gloves?*Product-Type=Gloves
Disallow /trousers?*Product-Type=Trousers
Disallow /body-armour?*Product-Type=Body-Armour
Disallow /accessories?*Product-Type=Accessories
Disallow /*Model%3D
Allow /*Brand%3D*%26Model%3D*
Allow /*Model%3D*%26Brand%3D*
Disallow /bagster/bagster-tank-covers?*Bike-Model=
Allow /bagster/bagster-tank-covers?*Bike-Make=*&Bike-Model=*
Allow /bagster/bagster-tank-covers?*Bike-Model=*&Bike-Make=*
Allow /bagster/bagster-tank-covers?*Bike-Make=*&Bike-Model=*&listing_page=*
Allow /bagster/bagster-tank-covers?*%3F*Bike-Model=*&Bike-Make=*&listing_page=*
Disallow /bagster/bagster-tank-covers?*listing_page=*
Disallow /bagster/bagster-tank-covers?*******************************************Year=*
Disallow /bagster/bagster-tank-covers?*******************************************Colour=*
Disallow /bagster/bagster-tank-covers?*******************************************Brand=Bagster*
Disallow /bagster/bagster-tank-covers?*******************************************Product-Type=Tank-Covers*
Disallow /bagster?*
Disallow /bagster/bagster-tank-bags?*Brand=Bagster*
Disallow /bagster/bagster-spare-parts?*Brand=Bagster*
Disallow /accessories?*Brand=Bagster
Disallow /bagster/bagster-tank-covers?************************************expand=26
Disallow /bagster/bagster-tank-covers?*expand=25
Allow /bagster/bagster-tank-covers?*Bike-Make=*&expand=25
Disallow /resources/jquery/jquery-facetednav.js*

baiduspider
baiduspider-video
baiduspider-image
sogou spider
youdaobot
moget
ichiro
naverbot
yeti
nutch
spiderbot
spiderbot/nutch-1.7
ahrefsbot
blexbot
blexbot/1.0

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.motolegends.com/sitemap_www_motolegends_com_index.xml
Back to top

Comments

  • Do not leave any line breaks below until 76
  • Normal ecommerce jazz
  • Longest string takes priority. This is necessary due to conflicting Allow rules:
  • Platform specific variables
  • Some of these are legacy and no longer used
  • Crawling rules to prevent weird things being crawled
  • "Jackets Motorcycle Jackets" etc.
  • Prevent Model overindexation
  • Only allow model URLs to be crawled when there's a brand selected
  • Prevent bagster overindexation
  • Rules:
  • ~ Don't show bagster model if no make is selected
  • ~ Don't allow crawling of paginated series unless both make and model selected
  • ~ Don't allow crawling of year, colour, brand, or product type
  • ~ Don't allow crawling of 'expand' for Model unless Make selected. Never allow crawling of Expand for Year
  • The above are very long strings so that they take priority over the allows
  • There are 3,000 bagster tank covers, all with the exact same content.
  • Disallow tank cover product pages and hope search engines properly rank filtered cats
  • Removed Support #320548 Disallow: /bagster/bagster-tank-covers/*.html
  • Prevent robots from using AJAX navigation (Nojs fallback is well optimised)
  • Ask pointless/mean spiders not to crawl the site
Back to top

Warnings

  • 2 invalid lines.
Back to top