Log In Sign Up

simplybe.co.uk
robots.txt

Robots Exclusion Standard data for simplybe.co.uk

Resource Scan

Scan Details

Site Domain simplybe.co.uk
Base Domain simplybe.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-08-22T11:24:37+00:00
Next Scan 2024-11-20T11:24:37+00:00

Last Successful Scan

Scanned2022-10-09T03:30:48+00:00
URL https://simplybe.co.uk/robots.txt
Redirect https://www.simplybe.co.uk/robots.txt
Redirect Domain www.simplybe.co.uk
Redirect Base simplybe.co.uk
Response IP 23.59.168.145, 23.59.168.179
Found Yes
Hash 882d728879c7564d746e8a8ade37b3f3d56573f76c90a28e5a75d2ecc5e92773
SimHash 31a35312abcf

Groups

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1.0

turnitinbot

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

*

Rule Path
Disallow /shop/ajax/*
Disallow /shop/ChangeTitle.servlet
Disallow /?referrerref=hpagelink
Disallow /shop/warranties/cover.action
Disallow /shop/policies/policiesView.action?finalTarget=policies_view&decoration=true
Disallow /shop/page?pageId=2401
Disallow /shop/product/details/ajax/
Disallow /shop/*/product/details/*ImageAnchor
Disallow /shop/product/details/*ImageAnchor
Disallow /shop/*/product/details/productVideoLink
Disallow /shop/product/details/productVideoLink
Disallow /shop/product/details/deliveryReturnsCopy
Disallow /shop/location/
Disallow /home.jsp
Disallow /shop/sitemap/advanced/
Disallow /shop/requestCatalogue.jsp
Disallow /shop/incentives/
Disallow /shop/contactus.jsp
Disallow /shop/request/catalogue/
Disallow /shop/viewbag/
Disallow /shop/qof/
Disallow /shop/emailsignup/
Disallow /shop/EmailCollectionController.servlet
Disallow /shop/product/details/guides.action
Disallow /shop/productreviews/writeSignedIn.action
Disallow /shop/signin/display.action
Disallow /shop/myaccount/AccountTransactionsOrders.action?
Disallow /dynamicPresentation
Disallow /shop/page/blog/show.action/search
Disallow /shop/verisign.jsp
Disallow /content/images/products/
Disallow /product/window/show.action
Disallow /?newtitle*
Disallow /?cm_re*
Disallow /?cm_decoration=true
Disallow /EmailCollectionController.servlet*
Disallow /EmailCollectionController.servlet*
Disallow /?combine%3F
Disallow /advancedSiteMap.jsp*
Disallow /advancedSiteMap_view*
Disallow /admin/*
Disallow /aclick/*
Disallow /acctreg_end.jsp
Disallow /aclick/*
Disallow /basket*
Disallow /checkout*
Disallow /Claim*
Disallow /contact_us.js*
Disallow /ContactusController.*
Disallow /dispatcher.*
Disallow /DynamicCoremetricProvider*
Disallow /dynamicPresentation/*
Disallow /financial_services.jsp*
Disallow /help.jsp*
Disallow /lpg_*
Disallow /productwindow.jsp*
Disallow /search_display.jsp*
Disallow /EmailCollectionController*
Disallow /favourites.jsp
Disallow /_Incapsula_Resource*
Disallow /ClaimCodes.servlet*
Disallow /*utm_customerid*
Disallow /shop/product/details/ajax/addPandpToBag.action*
Disallow /shop/json/mobileProductList.action*
Disallow /shop/viewbag/*
Disallow /shop/registration/jsonStringSecure.action*
Disallow /shop/s/*
Disallow /shop/product/details/productRange.action*
Disallow /shop/qof/start.action
Disallow /shop/*clearGroupUrl*
Disallow /shop/*facet-value-url*
Disallow /shop/*nextPage*
Disallow /shop/*previousPage*
Disallow /shop/*relUrl*
Disallow /shop/product/details/tapToZoom.action
Disallow *nofollowind%3Dtrue*
Disallow /shop/s?*
Disallow /shop/viewbag*
Disallow /shop/page//404/show.action*
Disallow /auth/realms/user/protocol/openid-connect/auth*
Disallow /shop/bag*
Disallow */Brand$
Disallow */Colour$
Disallow */Sizes$
Disallow */Categories$
Disallow */Design$
Disallow */Stone$
Disallow */Area$
Disallow */Metal$
Disallow */Outlet$
Disallow */useful-information$
Disallow */pages/free-format-directory$
Disallow */f/*/f/*
Disallow */c/*/c/*
Disallow */s/*/s/*
Disallow */p/*/p/*
Back to top

Comments

  • 2022-06-29 SH - updated version to cover de-risking phase 4 roll-out
  • LM Update 2018-02-24
  • Data from archive.org API extract 2019-12-11
  • Data from archive.org API extract 2019-12-18
  • Update: 2020-01-10
  • LM 2019-01-17
  • 2020-10-15 Blocking internal search again
  • 2020-10-29 LM
  • 2021-02-16 LM
  • 2021-12-07 SH
  • 2021-12-15 DH
  • 2022-05-31 Blocking internal search on nfe
  • 2021-06-21 LM - Keep after go-live
  • 2022-06-20 SH - Amended Disallowing view all facet urls
  • 2022-06-20 SH - Disallowing other pages
Back to top

Warnings

  • 27 invalid lines.
Back to top