Log In Sign Up

rexel.co.uk
robots.txt

Robots Exclusion Standard data for rexel.co.uk

Resource Scan

Scan Details

Site Domain rexel.co.uk
Base Domain rexel.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-11-12T05:41:24+00:00
Next Scan 2024-11-26T05:41:24+00:00

Last Successful Scan

Scanned2024-10-20T02:29:43+00:00
URL https://rexel.co.uk/robots.txt
Redirect https://www.rexel.co.uk/robots.txt
Redirect Domain www.rexel.co.uk
Redirect Base rexel.co.uk
Domain IPs 13.107.246.59
Redirect IPs 13.107.246.59
Response IP 13.107.246.59
Found Yes
Hash 754bd7ee7cc741e16dc12284ba124e98ff5f88ba116a001aebee623144126126
SimHash 518278576a50

Groups

*

Rule Path Comment
Disallow */uki/cart -
Disallow */uki/cart/* -
Disallow *_%3D* -
Disallow */*/c/brands -
Disallow *//c/ -
Disallow */_s/saveFulfillmentJson -
Disallow */addBulkProductItemsToCart -
Disallow */addBulkProductItemsToCartReplacementModel -
Disallow */addCustomerUserNboProductTocart -
Disallow */addObseleteReplacementProductItemsToCart -
Disallow */addProductsToCartFormConfigurator -
Disallow */addToCartFromUrl -
Disallow */addtoJoblist/* -
Disallow */c/.seo-collapse -
Disallow */**/c/browseCategoryAndBrandForMobile -
Disallow */checkout -
Disallow */checkout/* -
Disallow */closeGlobalPromotion -
Disallow */closeNotificationIcon -
Disallow *collapseTwo -
Disallow */combined.css* -
Disallow */combined.js* -
Disallow */compare/ -
Disallow */cookieCosent -
Disallow */cookieCosentToggler -
Disallow */deliveryDateSimulatorData.json -
Disallow */enableCookieConsent -
Disallow */erp/* -
Disallow */fast-register -
Disallow */findDefaultStore -
Disallow */findNearestStoreForAnonymous -
Disallow */footer/globalSites -
Disallow */GDPRCookieContainerComponentController/* -
Disallow */getBranchList* -
Disallow */getHtmlFormContent -
Disallow */getInspirationCompanyUrl -
Disallow */getJWTTokenAndUserDetails -
Disallow */getLimitedDistributionPickupZip -
Disallow */getNotificationMsgvaluesForLD -
Disallow */isQuantityVisibleForProducts -
Disallow */oauth/getOauthToken -
Disallow */getHtmlFormContent -
Disallow */getPagablePickupBranchesForAnonymous.json -
Disallow */getRegionList -
Disallow */getReplacementForObsoleteproduct -
Disallow */getTieredPricingMap -
Disallow */getJWTTokenAndUserDetails -
Disallow */isQuantitySectionVisible -
Disallow */isQuantityVisibleForProducts -
Disallow */login -
Disallow */login/* -
Disallow */logout -
Disallow */medias/*.pdf -
Disallow */mergeCart/* -
Disallow */mobilelogin* -
Disallow */mobileprivacy -
Disallow */mobileRegistration* -
Disallow */mockMedia/* -
Disallow */mostordered/getcategories -
Disallow */my-account -
Disallow */my-account/* -
Disallow */myaccountpopup -
Disallow */my-company/* -
Disallow */mysettings -
Disallow */newQuickorder/* -
Disallow */notification/* -
Disallow */oauth/getOauthToken -
Disallow */obseleteProductAddToCart -
Disallow */order-return/* -
Disallow */order-new-return/* -
Disallow */p/getEmailPDPpopup -
Disallow */p/getTieredPricing -
Disallow */p/slangKeyword -
Disallow */pf/getProductSelectedVariant -
Disallow */pf/productvariants -
Disallow *page%3D0 -
Disallow *page%3D-1 -
Disallow *pageSize%3D*%26 -
Disallow */perse/getNoResultsPage -
Disallow */prefillAddToCart -
Disallow */priceToolTip/* -
Disallow */product360view/* -
Disallow */productBranchStock -
Disallow */productBranchStock/* -
Disallow */punchoutCxmlCart/punchout -
Disallow */quickorder -
Disallow */recommendedCategories.ajax -
Disallow */register -
Disallow */register/* -
Disallow */removeChangeRegionPopup -
Disallow */removeGDPRLegalNotification -
Disallow */removePCIComplianceNotification -
Disallow */replacementProduct -
Disallow */sales/* -
Disallow */search/* -
Disallow */search* -
Disallow */search/searchCategory -
Disallow */updatelanguage -
Disallow */updateRegion -
Disallow */validateUpdateRegion -
Disallow */view/TabbedNavigationComponentController/updateTabIndex -
Disallow */p/*/zoomImages -
Disallow *%7B0%7D -
Disallow *%7B0%7D -
Disallow */cms/getPDPContent.json?productCode=* -
Allow *%3A*%3A*/p/* -
Disallow /search*.com -
Disallow /search*.net -
Disallow *isOnlyProductIds%3D%26isOrderForm%3D%26skuIndex%3D%26isCreateOrderForm* -
Disallow */c/*displayProductList* -
Disallow *q%3D%3Arelevance*%26filter%3D*%3A* old + new
Disallow *relevance%26filter%3D*%3A* old + new
Disallow *filter%3D*%3A* -
Disallow *filter%3D*%3A*~* -
Disallow *filter%3D*%3A*%3B*%3A* -
Allow *filter%3DbrandCode%3A* -
Disallow *filter%3DbrandCode%3A*~ -
Disallow *filter%3DbrandCode%3A*%3B* -
Disallow */brands/*/c/*filter%3DbrandCode%3A* -
Allow */brands/*/c/*filter%3DcategoryCode%3A* -
Disallow */brands/*/c/*filter%3DcategoryCode%3A*~ -
Disallow */brands/*/c/*filter%3DcategoryCode%3A*%3B* -
Disallow *main-*.js -
Disallow *scripts-*.js -
Disallow *polyfills-*.js -
Disallow *runtime-*.js -
Disallow *%3Arelevance%3A*%3A* -
Disallow *%3Arelevance%3A*%3A* -
Disallow */c/*__* -
Disallow */c/*/*productbranchstock* -
Disallow *page%3D-1* -
Disallow */c/*?q=%3Arelevance&text=&filter=&aliasname=&partialSearch= -
Disallow */root-category/*/c/*relevance%3Acategory* -
Disallow */brands/*/c/*relevance%3Abrand* -
Allow */root-category/*/c/*q%3D%3Arelevance%3Abrand%3A* -
Disallow */root-category/*/c/*q%3D%3Arelevance%3Abrand%3A*%3A -
Disallow */root-category/*/c/*q%3D%3Arelevance%3Abrand%3A*page%3D-1* -
Allow */brands/*/c/*q%3D%3Arelevance%3Acategory%3A* -
Disallow */brands/*/c/*q%3D%3Arelevance%3Acategory%3A*%3A -
Disallow */brands/*/c/*q%3D%3Arelevance%3Acategory%3A*page%3D-1* -

bytespider

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.rexel.co.uk/uki/sitemap.xml
Back to top

Comments

  • cms related
  • testing
  • allow odd PDP combinations
  • spam pattern
  • site specific blocks
  • old pagination facets - params - tests
  • PERSE
  • Limit PLP Facet Crawling
  • Disallow facets but those allowed elsewhere
  • disallow other
  • disallow facet removal
  • disallow category facet on category pages
  • Disallow brand facet on brand pages
  • Allow / Disallow Brand facets for category pages
  • Allow / Disallow Category Facets for Brand pages
  • v20240523
Back to top