Log In Sign Up

rexel.com.au
robots.txt

Robots Exclusion Standard data for rexel.com.au

Resource Scan

Scan Details

Site Domain rexel.com.au
Base Domain rexel.com.au
Scan Status Ok
Last Scan2024-09-17T01:05:50+00:00
Next Scan 2024-10-01T01:05:50+00:00

Last Scan

Scanned2024-09-17T01:05:50+00:00
URL https://rexel.com.au/robots.txt
Domain IPs 168.138.3.92
Response IP 168.138.3.92
Found Yes
Hash b7172eb81aae825b74ed13483b37f995a26a688f26cd4ea734a078028caf55e2
SimHash 85033a636e10

Groups

*

Rule Path
Disallow */are/cart
Disallow */are/cart/*
Disallow *_%3D*
Disallow */*/c/brands
Disallow *//c/
Disallow */_s/saveFulfillmentJson
Disallow */addBulkProductItemsToCart
Disallow */addBulkProductItemsToCartReplacementModel
Disallow */addCustomerUserNboProductTocart
Disallow */addObseleteReplacementProductItemsToCart
Disallow */addProductsToCartFormConfigurator
Disallow */addToCartFromUrl
Disallow */addtoJoblist/*
Disallow */c/.seo-collapse
Disallow */**/c/browseCategoryAndBrandForMobile
Disallow */checkout
Disallow */checkout/*
Disallow */closeGlobalPromotion
Disallow */closeNotificationIcon
Disallow *collapseTwo
Disallow */combined.css*
Disallow */combined.js*
Disallow */compare/
Disallow */cookieCosent
Disallow */cookieCosentToggler
Disallow */deliveryDateSimulatorData.json
Disallow */enableCookieConsent
Disallow */erp/*
Disallow */fast-register
Disallow */findDefaultStore
Disallow */findNearestStoreForAnonymous
Disallow */footer/globalSites
Disallow */GDPRCookieContainerComponentController/*
Disallow */getBranchList*
Disallow */getHtmlFormContent
Disallow */getInspirationCompanyUrl
Disallow */getJWTTokenAndUserDetails
Disallow */getLimitedDistributionPickupZip
Disallow */getNotificationMsgvaluesForLD
Disallow */isQuantityVisibleForProducts
Disallow */oauth/getOauthToken
Disallow */getHtmlFormContent
Disallow */getPagablePickupBranchesForAnonymous.json
Disallow */getRegionList
Disallow */getReplacementForObsoleteproduct
Disallow */getTieredPricingMap
Disallow */getJWTTokenAndUserDetails
Disallow */isQuantitySectionVisible
Disallow */isQuantityVisibleForProducts
Disallow */login
Disallow */login/*
Disallow */logout
Disallow */mergeCart/*
Disallow */mobilelogin*
Disallow */mobileprivacy
Disallow */mobileRegistration*
Disallow */mockMedia/*
Disallow */mostordered/getcategories
Disallow */my-account
Disallow */my-account/*
Disallow */myaccountpopup
Disallow */my-company/*
Disallow */mysettings
Disallow */newQuickorder/*
Disallow */notification/*
Disallow */oauth/getOauthToken
Disallow */obseleteProductAddToCart
Disallow */order-return/*
Disallow */order-new-return/*
Disallow */p/getEmailPDPpopup
Disallow */p/getTieredPricing
Disallow */p/slangKeyword
Disallow */pf/getProductSelectedVariant
Disallow */pf/productvariants
Disallow *page%3D0
Disallow *page%3D-1
Disallow *pageSize%3D*%26
Disallow */perse/getNoResultsPage
Disallow */prefillAddToCart
Disallow */priceToolTip/*
Disallow */product360view/*
Disallow */productBranchStock
Disallow */productBranchStock/*
Disallow */punchoutCxmlCart/punchout
Disallow */quickorder
Disallow */recommendedCategories.ajax
Disallow */register
Disallow */register/*
Disallow */removeChangeRegionPopup
Disallow */removeGDPRLegalNotification
Disallow */removePCIComplianceNotification
Disallow */replacementProduct
Disallow */sales/*
Disallow */updatelanguage
Disallow */updateRegion
Disallow */validateUpdateRegion
Disallow */view/TabbedNavigationComponentController/updateTabIndex
Disallow */p/*/zoomImages
Disallow *%7B0%7D
Disallow *%7B0%7D
Disallow */cms/getPDPContent.json?productCode=*
Disallow */_ui/addons/*
Disallow */_ui/mobile/common/*
Disallow */_ui/resonsive/common/js/*
Disallow */perse/getNoResultsPage
Disallow */search*
Disallow */_ui/angular/*
Allow *%3A*%3A*/p/*
Disallow /search*.com
Disallow /search*.net
Disallow *%3Arelevance%3A*%3A*
Disallow *%3Arelevance%3A*%3A*
Disallow *%3A*%3A*
Disallow *%3A*%3A*
Disallow */c/*__*
Disallow */c/*/*productbranchstock*
Disallow *page%3D-1*
Disallow */c/*?q=%3Arelevance&text=&filter=&aliasname=&partialSearch=
Disallow */category/*/c/*relevance%3Acategory*
Disallow */brands/*/c/*relevance%3Abrand*
Disallow */category/*/c/*%3Arelevance%3A*%3A*%3A*%3A*
Disallow */brand/*/c/*%3Arelevance%3A*%3A*%3A*3A*
Allow */category/*/c/*q%3D%3Arelevance%3Abrand%3A*
Disallow */category/*/c/*q%3D%3Arelevance%3Abrand%3A*%3A
Disallow */category/*/c/*q%3D%3Arelevance%3Abrand%3A*page%3D-1*
Disallow */category/*/c/*q%3D*%3Arelevance%3Abrand%3A*%3A
Allow */brands/*/c/*%3Arelevance%3Acategory%3A*
Disallow */brands/*/c/*q%3D%3Arelevance%3Acategory%3A*%3A
Disallow */brands/*/c/*q%3D%3Arelevance%3Acategory%3A*page%3D-1*

scrapy

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.2

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.rexel.com.au/are/sitemap.xml
Back to top

Comments

  • cms related
  • testing
  • allow odd PDP combinations
  • spam pattern
  • Limit PLP Facet Crawling
  • Disallow facets but those allowed elsewhere
  • disallow other
  • disallow facet removal
  • disallow category facet on category pages
  • Disallow brand facet on brand pages
  • disallow multi facet selections on category and brand
  • Allow / Disallow Brand facets for category pages
  • Allow / Disallow Category Facets for Brand pages
  • v20230810
Back to top