Log In Sign Up

crazyclearance.co.uk
robots.txt

Robots Exclusion Standard data for crazyclearance.co.uk

Resource Scan

Scan Details

Site Domain crazyclearance.co.uk
Base Domain crazyclearance.co.uk
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-03-18T11:00:15+00:00
Next Scan 2024-06-16T11:00:15+00:00

Last Successful Scan

Scanned2022-10-30T21:17:22+00:00
URL https://crazyclearance.co.uk/robots.txt
Redirect https://www.crazyclearance.co.uk/robots.txt
Redirect Domain www.crazyclearance.co.uk
Redirect Base crazyclearance.co.uk
Response IP 23.52.171.241, 23.59.168.154
Found Yes
Hash 08fed7ce41382f7117ff98d687994fe5c32a7f58a8aeeadff4d1c8bd2b662eaa
SimHash 31ef54722bed

Groups

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1.0

turnitinbot

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

*

Rule Path
Disallow /shop/ajax/*
Disallow /shop/ChangeTitle.servlet
Disallow /?referrerref=hpagelink
Disallow /shop/warranties/cover.action
Disallow /shop/policies/policiesView.action?finalTarget=policies_view&decoration=true
Disallow /shop/page?pageId=2401
Disallow /shop/product/details/ajax/
Disallow /shop/*/product/details/*ImageAnchor
Disallow /shop/product/details/*ImageAnchor
Disallow /shop/*/product/details/productVideoLink
Disallow /shop/product/details/productVideoLink
Disallow /shop/product/details/deliveryReturnsCopy
Disallow /shop/location/
Disallow /home.jsp
Disallow /shop/sitemap/advanced/
Disallow /shop/requestCatalogue.jsp
Disallow /shop/incentives/
Disallow /shop/contactus.jsp
Disallow /shop/request/catalogue/
Disallow /shop/viewbag/
Disallow /shop/qof/
Disallow /shop/emailsignup/
Disallow /shop/EmailCollectionController.servlet
Disallow /shop/product/details/guides.action
Disallow /shop/productreviews/writeSignedIn.action
Disallow /shop/signin/display.action
Disallow /shop/myaccount/AccountTransactionsOrders.action?
Disallow /dynamicPresentation
Disallow /shop/page/blog/show.action/search
Disallow /shop/verisign.jsp
Disallow /content/images/products/
Disallow /product/window/show.action
Disallow /?newtitle*
Disallow /?cm_re*
Disallow /?cm_decoration=true
Disallow /EmailCollectionController.servlet*
Disallow /EmailCollectionController.servlet*
Disallow /?combine%3F
Disallow /advancedSiteMap.jsp*
Disallow /advancedSiteMap_view*
Disallow /admin/*
Disallow /aclick/*
Disallow /acctreg_end.jsp
Disallow /aclick/*
Disallow /basket*
Disallow /checkout*
Disallow /Claim*
Disallow /contact_us.js*
Disallow /ContactusController.*
Disallow /dispatcher.*
Disallow /DynamicCoremetricProvider*
Disallow /dynamicPresentation/*
Disallow /financial_services.jsp*
Disallow /help.jsp*
Disallow /lpg_*
Disallow /productwindow.jsp*
Disallow /search_display.jsp*
Disallow /EmailCollectionController*
Disallow /favourites.jsp
Disallow /_Incapsula_Resource*
Disallow /ClaimCodes.servlet*
Disallow /*utm_customerid*
Disallow /shop/product/details/ajax/addPandpToBag.action*
Disallow /shop/json/mobileProductList.action*
Disallow /shop/viewbag/*
Disallow /shop/registration/jsonStringSecure.action*
Disallow /shop/s/*
Disallow /shop/product/details/productRange.action*
Disallow /shop/qof/start.action
Disallow /shop/*clearGroupUrl*
Disallow /shop/*facet-value-url*
Disallow /shop/*nextPage*
Disallow /shop/*previousPage*
Disallow /shop/*relUrl*
Disallow /shop/product/details/tapToZoom.action
Disallow *nofollowind%3Dtrue*
Disallow */store_finder*/show.action
Back to top

Comments

  • LM Update 2018-02-24
  • Data from archive.org API extract 2019-12-11
  • Data from archive.org API extract 2019-12-18
  • Update: 2020-01-10
  • LM 2019-01-17
  • 2020-10-15 Blocking internal search again
  • 2020-10-29 LM
  • 2021-02-16 LM
  • 2021-12-07 SH
  • 2021-12-15 DH
  • 2022-09-02 SH# 2022-09-02 SH
Back to top