Log In Sign Up

dreams.co.uk
robots.txt

Robots Exclusion Standard data for dreams.co.uk

Resource Scan

Scan Details

Site Domain dreams.co.uk
Base Domain dreams.co.uk
Scan Status Ok
Last Scan2024-09-19T18:02:27+00:00
Next Scan 2024-10-19T18:02:27+00:00

Last Scan

Scanned2024-09-19T18:02:27+00:00
URL https://dreams.co.uk/robots.txt
Redirect https://www.dreams.co.uk/robots.txt
Redirect Domain www.dreams.co.uk
Redirect Base dreams.co.uk
Domain IPs 104.22.60.121, 104.22.61.121, 172.67.5.196
Redirect IPs 104.22.60.121, 104.22.61.121, 172.67.5.196
Response IP 104.22.61.121
Found Yes
Hash d9ff03f65d41f09fb6ffb4561a290ce5893090dd553187d348a540d4b0d4da38
SimHash 3e56d772ccfe

Groups

*

Rule Path
Disallow /*/nonCacheVerify
Disallow /_s/*
Disallow /cart/miniCart/
Disallow /checkout
Disallow /my-account/*
Disallow /search
Disallow /shortlist/*
Disallow /t/*
Disallow /recently-viewed/*
Disallow */resultsandfacets
Disallow */dimensionSelector

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.dreams.co.uk/sitemap.xml
sitemap https://www.dreams.co.uk/sleep-matters-club/sitemap_index.xml
Back to top

Comments

  • For all robots
  • Block access to specific groups of pages
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Delay rogerbot https://moz.com/help/moz-procedures/crawlers/rogerbot
  • SMC
Back to top