Log In Sign Up

westburne.ca
robots.txt

Robots Exclusion Standard data for westburne.ca

Resource Scan

Scan Details

Site Domain westburne.ca
Base Domain westburne.ca
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-11-11T15:45:35+00:00
Next Scan 2026-01-10T15:45:35+00:00

Last Successful Scan

Scanned2025-09-06T00:30:54+00:00
URL https://westburne.ca/robots.txt
Redirect https://www.westburne.ca/robots.txt
Redirect Domain www.westburne.ca
Redirect Base westburne.ca
Domain IPs 13.107.253.59
Redirect IPs 13.107.246.59, 2620:1ec:bdf::59
Response IP 13.107.246.59
Found Yes
Hash be3c9d2656578f6bdc7c2b6d6a272a6fbac51db26f6330da87679d6109372693
SimHash 95425a636e10

Groups

*

Rule Path
Disallow */cwr/cart
Disallow */cwr/cart/*
Disallow *_%3D*
Disallow */*/c/brands
Disallow *//c/
Disallow */_s/saveFulfillmentJson
Disallow */addBulkProductItemsToCart
Disallow */addBulkProductItemsToCartReplacementModel
Disallow */addCustomerUserNboProductTocart
Disallow */addObseleteReplacementProductItemsToCart
Disallow */addProductsToCartFormConfigurator
Disallow */addToCartFromUrl
Disallow */addtoJoblist/*
Disallow */c/.seo-collapse
Disallow */**/c/browseCategoryAndBrandForMobile
Disallow */checkout
Disallow */checkout/*
Disallow */closeGlobalPromotion
Disallow */closeNotificationIcon
Disallow *collapseTwo
Disallow */combined.css*
Disallow */combined.js*
Disallow */compare/
Disallow */cookieCosent
Disallow */cookieCosentToggler
Disallow */enableCookieConsent
Disallow */erp/*
Disallow */fast-register
Disallow */findDefaultStore
Disallow */findNearestStoreForAnonymous
Disallow */footer/globalSites
Disallow */GDPRCookieContainerComponentController/*
Disallow */getBranchList*
Disallow */getHtmlFormContent
Disallow */getInspirationCompanyUrl
Disallow */getJWTTokenAndUserDetails
Disallow */getLimitedDistributionPickupZip
Disallow */getNotificationMsgvaluesForLD
Disallow */isQuantityVisibleForProducts
Disallow */oauth/getOauthToken
Disallow */getPagablePickupBranchesForAnonymous.json
Disallow */getRegionList
Disallow */getReplacementForObsoleteproduct
Disallow */getTieredPricingMap
Disallow */isQuantitySectionVisible
Disallow */login
Disallow */login/*
Disallow */logout
Disallow */mergeCart/*
Disallow */mobilelogin*
Disallow */mobileprivacy
Disallow */mobileRegistration*
Disallow */mockMedia/*
Disallow */my-account
Disallow */my-account/*
Disallow */myaccountpopup
Disallow */my-company/*
Disallow */mysettings
Disallow */notification/*
Disallow */oauth/getOauthToken
Disallow */obseleteProductAddToCart
Disallow */order-return/*
Disallow */order-new-return/*
Disallow */p/getEmailPDPpopup
Disallow */p/getTieredPricing
Disallow */p/slangKeyword
Disallow *page%3D0
Disallow *page%3D-1
Disallow *pageSize%3D*%26
Disallow */prefillAddToCart
Disallow */priceToolTip/*
Disallow */product360view/*
Disallow */productBranchStock
Disallow */productBranchStock/*
Disallow */quickorder
Disallow */recommendedCategories.ajax
Disallow */register
Disallow */register/*
Disallow */removeChangeRegionPopup
Disallow */removeGDPRLegalNotification
Disallow */removePCIComplianceNotification
Disallow */replacementProduct
Disallow */sales/*
Disallow */updatelanguage
Disallow */updateRegion
Disallow */validateUpdateRegion
Disallow */view/TabbedNavigationComponentController/updateTabIndex
Disallow */p/*/zoomImages
Disallow *%7B0%7D
Disallow *%7B0%7D
Disallow */cms/getPDPContent.json?productCode=*
Disallow */_ui/addons/*
Disallow */_ui/mobile/common/*
Disallow */_ui/resonsive/common/js/*
Disallow */perse/getNoResultsPage
Disallow */search*
Disallow */_ui/angular/*
Allow *%3A*%3A*/p/*
Allow */p/*/*
Disallow /search*.com
Disallow /search*.net
Disallow */medias/*
Disallow *%3Arelevance%3A*%3A*
Disallow *%3Arelevance%3A*%3A*
Disallow *%3A*%3A*
Disallow *%3A*%3A*
Disallow */c/*__*
Disallow */c/*/*productbranchstock*
Disallow *page%3D-1*
Disallow */c/*?q=%3Arelevance&text=&filter=&aliasname=&partialSearch=
Disallow */Category/*/c/*relevance%3Acategory*
Disallow */Brand/*/c/*relevance%3Abrand*
Allow */Category/*/c/*q%3D%3Arelevance%3Abrand%3A*
Disallow */Category/*/c/*q%3D%3Arelevance%3Abrand%3A*%3A
Disallow */Category/*/c/*q%3D%3Arelevance%3Abrand%3A*page%3D-1*
Allow */Brand/*/c/*q%3D%3Arelevance%3Acategory%3A*
Disallow */Brand/*/c/*q%3D%3Arelevance%3Acategory%3A*%3A
Disallow */Brand/*/c/*q%3D%3Arelevance%3Acategory%3A*page%3D-1*

scrapy

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.westburne.ca/cwr/sitemap.xml
Back to top

Comments

  • cms related
  • testing
  • spam pattern
  • site specific blocks
  • Limit PLP Facet Crawling
  • Disallow facets but those allowed elsewhere
  • disallow other
  • disallow facet removal
  • disallow category facet on category pages
  • Disallow brand facet on brand pages
  • Allow / Disallow Brand facets for category pages
  • Allow / Disallow Category Facets for Brand pages
  • v20231017
Back to top