it.boohoo.com
robots.txt

Robots Exclusion Standard data for it.boohoo.com

Resource Scan

Scan Details

Site Domain it.boohoo.com
Base Domain boohoo.com
Scan Status Ok
Last Scan2024-11-08T06:17:52+00:00
Next Scan 2024-11-22T06:17:52+00:00

Last Scan

Scanned2024-11-08T06:17:52+00:00
URL https://it.boohoo.com/robots.txt
Domain IPs 23.59.168.97, 2600:1413:b000:1b::17d7:708, 2600:1413:b000:1b::17d7:715
Response IP 23.59.168.97
Found Yes
Hash 365daf44e09b2b9d2cde789d0233a67485c95457c07c5f8f35d5626c929a1876
SimHash 22619f0ea2e5

Groups

*

Rule Path
Disallow /*placeholder?*
Disallow /account-edit*
Disallow /preferences-edit*
Disallow /checkout-login*
Disallow /order-history*
Disallow /order-confirmation*
Disallow /on/demandware.store/*
Disallow /on/demandware.store/*/Tile-UpdateVariantTile*
Disallow /on/demandware.store/*/Search-ShowAjax*
Allow /on/demandware.store/*/Product-Show?pid=*
Disallow /*/MPP*.html
Disallow /*/BPP*.html
Disallow /*/AGG*.html
Disallow /*/BGG*.html
Disallow */Wishlist-Add*
Disallow *wishlist-add*
Disallow */Product-Variation*
Disallow /*sz%3D16%26start%3D*
Disallow /*srule%3D*
Disallow /*pmin%3D*
Disallow /*pmax%3D*
Disallow /*prefn*
Disallow /*prefv*
Disallow /*%7C*
Allow /search?q=christmas%2Bjumper
Allow /search?q=pride
Allow /search?q=chelsea%2Bboots
Allow /search?q=Blankets
Allow /search?q=face%20masks&start=60&sz=60
Allow /search?q=recycled
Allow /search?q=leaf%2Bbikini
Allow /search?q=workout%2Bleggings
Allow /search?q=summer%2Bdresses
Allow /search?q=Palmer%27s
Allow /search?q=bride
Allow /search?q=fur%2Bcoat
Allow /search?q=Dresses&prefn1=color&prefv1=Gold
Disallow /search?q=Dresses&prefn1=color&prefv1=Gold&*
Disallow /search?q=Dresses&prefn1=color&prefv1=Gold%7C*
Allow /search?q=Men%2BChristmas%2Bjumpers
Allow /search?q=boohoo%2Bhis%2Bher
Allow /search?q=short%20dresses
Allow /search?q=long%2Bskirt
Allow /search?q=plaid
Allow /page/search?q=Jennifer%2BSatin%2BPlaysuit
Allow /page/search?q=mesh
Allow /page/search?q=sports%2Bcrop
Allow /page/search?q=playsuit
Allow /page/search?q=hats
Allow /page/search?q=black%2Bcut%2Bout%2Bdress
Allow /page/search?q=sports%2Bleggings
Allow /page/search?q=crop%2Btops
Allow /page/search?q=jumper
Allow /page/search?q=CHRISTMAS%2BJUMPER
Allow /page/search?q=fluffy%2Bbag
Allow /page/search?q=jumper%2Bdress
Allow /search?cgid=body-jewelry
Allow /search?q=bodysuit&prefn1=color&prefv1=White
Allow /search?cgid=boohooontheroad-shop
Allow /search?cgid=pepsi-x-boohoo
Allow /search?q=fashion%20mask
Allow /search?cgid=celeb-style
Allow /search?q=prom%2Bdresses
Allow /search?q=culotte
Allow /search?cgid=all-clothes
Allow /search?cgid=western
Allow /search?q=cropped%2Bhoodie
Allow /search?q=floral
Allow /search?q=fashion%2Bmask
Allow /search?cgid=jackets-coats
Allow /search?cgid=clothing
Allow /search?q=bomber%2Bjacket
Allow /search?cgid=all-that-glitters
Allow /search?q=casual%20dress
Allow /search?q=christmas%2Bjumpers
Allow /search?q=know%2Byour%2Blemons
Allow /search?q=white%2Bblazer%2Bwomen
Allow /search?q=tartan
Allow /search?q=velours
Allow /search?q=western
Allow /search?cgid=charlixcxforboohoo-shop
Disallow /*search*
Allow /*prefn1%3Dcolor%26prefv1%3D*
Allow /*prefn1%3Dsize%26prefv1%3D*
Disallow /*prefn*%3Dcolor%26prefv*%3D*%7C*
Disallow /*prefn*%3Dsize%26prefv*%3D*%7C*
Disallow /*search*prefn*%3D*%26prefv*%3D*
Disallow /*prefn*%3D*%26prefv*%3D*start%3D*
Disallow /*prefn*%3D*%26prefv*%3D*srule%3D*
Allow /on/demandware.static/*/*.js
Allow /on/demandware.static/*/*.css
Allow /on/demandware.static/*/*.json
Allow /on/demandware.static/*/*.jpg
Allow /on/demandware.static/*/*.jpeg
Allow /on/demandware.static/*/*.webp
Allow /on/demandware.static/*/*.woff2
Allow /on/demandware.static/*/*.png
Allow /on/demandware.static/*/*.gif
Allow /on/demandware.static/*/*.svg

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0.2

Other Records

Field Value
sitemap https://it.boohoo.com/sitemap_index.xml
sitemap https://es.boohoo.com/sitemap_index.xml

Comments

  • unrelated URLS
  • Pages
  • Product Filter
  • Ordering & Product per page
  • Number of product per page
  • Order By
  • Price
  • Faceted Navigation
  • UK & ALL Search
  • US Search
  • AU Search
  • IE Search
  • FR Search
  • Search
  • Handle Execption for colour/size attribute use as internal link
  • Additional Rules to handle exception
  • Ensure no Static Ressources is blocked
  • Crawl Delay - 5 URL max per second