Log In Sign Up

denmans.co.uk
robots.txt

Robots Exclusion Standard data for denmans.co.uk

Resource Scan

Scan Details

Site Domain denmans.co.uk
Base Domain denmans.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-07-01T04:44:26+00:00
Next Scan 2024-07-08T04:44:26+00:00

Last Successful Scan

Scanned2024-06-16T04:43:47+00:00
URL https://denmans.co.uk/robots.txt
Redirect https://www.denmans.co.uk/robots.txt
Redirect Domain www.denmans.co.uk
Redirect Base denmans.co.uk
Domain IPs 13.107.246.73
Redirect IPs 13.107.246.68
Response IP 13.107.246.59
Found Yes
Hash 5adc9d1923fb98a0de57c5cc2782593182f8f2e6a3e00b083e9390c92bda258f
SimHash d1825a576a12

Groups

*

Rule Path Comment
Disallow */den/cart -
Disallow */den/cart/* -
Disallow *_%3D* -
Disallow */*/c/brands -
Disallow *//c/ -
Disallow */_s/saveFulfillmentJson -
Disallow */addBulkProductItemsToCart -
Disallow */addBulkProductItemsToCartReplacementModel -
Disallow */addCustomerUserNboProductTocart -
Disallow */addObseleteReplacementProductItemsToCart -
Disallow */addProductsToCartFormConfigurator -
Disallow */addToCartFromUrl -
Disallow */addtoJoblist/* -
Disallow */c/.seo-collapse -
Disallow */**/c/browseCategoryAndBrandForMobile -
Disallow */checkout -
Disallow */checkout/* -
Disallow */closeGlobalPromotion -
Disallow */closeNotificationIcon -
Disallow *collapseTwo -
Disallow */combined.css* -
Disallow */combined.js* -
Disallow */compare/ -
Disallow */cookieCosent -
Disallow */cookieCosentToggler -
Disallow */deliveryDateSimulatorData.json -
Disallow */enableCookieConsent -
Disallow */erp/* -
Disallow */fast-register -
Disallow */findDefaultStore -
Disallow */findNearestStoreForAnonymous -
Disallow */footer/globalSites -
Disallow */GDPRCookieContainerComponentController/* -
Disallow */getBranchList* -
Disallow */getHtmlFormContent -
Disallow */getInspirationCompanyUrl -
Disallow */getJWTTokenAndUserDetails -
Disallow */getLimitedDistributionPickupZip -
Disallow */getNotificationMsgvaluesForLD -
Disallow */isQuantityVisibleForProducts -
Disallow */oauth/getOauthToken -
Disallow */getHtmlFormContent -
Disallow */getPagablePickupBranchesForAnonymous.json -
Disallow */getRegionList -
Disallow */getReplacementForObsoleteproduct -
Disallow */getTieredPricingMap -
Disallow */getJWTTokenAndUserDetails -
Disallow */isQuantitySectionVisible -
Disallow */isQuantityVisibleForProducts -
Disallow */login -
Disallow */login/* -
Disallow */logout -
Disallow */medias/*.pdf -
Disallow */mergeCart/* -
Disallow */mobilelogin* -
Disallow */mobileprivacy -
Disallow */mobileRegistration* -
Disallow */mockMedia/* -
Disallow */mostordered/getcategories -
Disallow */my-account -
Disallow */my-account/* -
Disallow */myaccountpopup -
Disallow */my-company/* -
Disallow */mysettings -
Disallow */newQuickorder/* -
Disallow */notification/* -
Disallow */oauth/getOauthToken -
Disallow */obseleteProductAddToCart -
Disallow */order-return/* -
Disallow */order-new-return/* -
Disallow */p/getEmailPDPpopup -
Disallow */p/getTieredPricing -
Disallow */p/slangKeyword -
Disallow */pf/getProductSelectedVariant -
Disallow */pf/productvariants -
Disallow *page%3D0 -
Disallow *page%3D-1 -
Disallow *pageSize%3D*%26 -
Disallow */perse/getNoResultsPage -
Disallow */prefillAddToCart -
Disallow */priceToolTip/* -
Disallow */product360view/* -
Disallow */productBranchStock -
Disallow */productBranchStock/* -
Disallow */punchoutCxmlCart/punchout -
Disallow */quickorder -
Disallow */recommendedCategories.ajax -
Disallow */register -
Disallow */register/* -
Disallow */removeChangeRegionPopup -
Disallow */removeGDPRLegalNotification -
Disallow */removePCIComplianceNotification -
Disallow */replacementProduct -
Disallow */sales/* -
Disallow */search/* -
Disallow */search* -
Disallow */search/searchCategory -
Disallow */updatelanguage -
Disallow */updateRegion -
Disallow */validateUpdateRegion -
Disallow */view/TabbedNavigationComponentController/updateTabIndex -
Disallow */p/*/zoomImages -
Disallow *%7B0%7D -
Disallow *%7B0%7D -
Disallow */cms/getPDPContent.json?productCode=* -
Allow *%3A*%3A*/p/* -
Allow -
Disallow /search*.com -
Disallow /search*.net -
Disallow *isOnlyProductIds%3D%26isOrderForm%3D%26skuIndex%3D%26isCreateOrderForm* -
Disallow */c/*displayProductList* -
Disallow *q%3D%3Arelevance*%26filter%3D*%3A* old + new
Disallow *relevance%26filter%3D*%3A* old + new
Allow *filter%3D*%3A* -
Disallow *filter%3D*%3A*~* -
Disallow *filter%3D*%3A*%3B*%3A* -
Disallow *main-*.js -
Disallow *scripts-*.js -
Disallow *polyfills-*.js -
Disallow *runtime-*.js -
Disallow *%3Arelevance%3A*%3A* -
Disallow *%3Arelevance%3A*%3A* -
Disallow */c/*__* -
Disallow */c/*/*productbranchstock* -
Disallow *page%3D-1* -
Disallow */c/*?q=%3Arelevance&text=&filter=&aliasname=&partialSearch= -
Disallow */root-category/*/c/*relevance%3Acategory* -
Disallow */brands/*/c/*relevance%3Abrand* -
Allow */root-category/*/c/*q%3D%3Arelevance%3Abrand%3A* -
Disallow */root-category/*/c/*q%3D%3Arelevance%3Abrand%3A*%3A -
Disallow */root-category/*/c/*q%3D%3Arelevance%3Abrand%3A*page%3D-1* -
Allow */brands/*/c/*q%3D%3Arelevance%3Acategory%3A* -
Disallow */brands/*/c/*q%3D%3Arelevance%3Acategory%3A*%3A -
Disallow */brands/*/c/*q%3D%3Arelevance%3Acategory%3A*page%3D-1* -

bytespider

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.denmans.co.uk/den/sitemap.xml
Back to top

Comments

  • cms related
  • testing
  • allow odd PDP combinations
  • spam pattern
  • site specific blocks
  • old pagination facets - params - tests
  • PERSE - ooe version
  • Limit PLP Facet Crawling
  • Disallow facets but those allowed elsewhere
  • disallow other
  • disallow facet removal
  • disallow category facet on category pages
  • Disallow brand facet on brand pages
  • Allow / Disallow Brand facets for category pages
  • Allow / Disallow Category Facets for Brand pages
  • v20240523
Back to top