aesop.com
robots.txt
Robots Exclusion Standard data for aesop.com
Resource Scan
Scan Details
| Site Domain | aesop.com |
| Base Domain | aesop.com |
| Scan Status | Ok |
| Last Scan | 2026-02-03T01:02:53+00:00 |
| Next Scan | 2026-03-05T01:02:53+00:00 |
Last Scan
| Scanned | 2026-02-03T01:02:53+00:00 |
| URL | https://aesop.com/robots.txt |
| Redirect | https://www.aesop.com/robots.txt |
| Redirect Domain | www.aesop.com |
| Redirect Base | aesop.com |
| Domain IPs | 104.18.43.80, 172.64.144.176, 2606:4700:4405::6812:2b50, 2a06:98c1:3101::ac40:90b0 |
| Redirect IPs | 104.18.43.80, 172.64.144.176, 2606:4700:4405::6812:2b50, 2a06:98c1:3101::ac40:90b0 |
| Response IP | 104.18.43.80 |
| Found | Yes |
| Hash | 750e582bfcdef9c776fe2dc712bd1f52c889f10946091d23515d806848f45929 |
| SimHash | e33c9c77b277 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /*bvstate |
| Disallow | /cart |
| Disallow | /account |
| Disallow | /register |
| Disallow | /shopping-bag |
| Disallow | /Address-List |
| Disallow | /checkout-start |
| Disallow | /Order-History |
| Disallow | */Search-* |
| Disallow | /Wishlist-Show |
| Disallow | /search |
| Allow | /on/demandware.static/Sites-aesop-us-Site/Sites-aesop-us-Library/default/ |
| Disallow | /on/demandware.static/Sites-aesop-us-Site/Sites-aesop-us-Library/default/v* |
| Disallow | /*prefn1%3D |
| Disallow | /*prefn2%3D |
| Disallow | /*prefn3%3D |
| Disallow | /*prefn4%3D |
| Disallow | /*prefv1%3D |
| Disallow | /*prefv2%3D |
| Disallow | /*prefv3%3D |
| Disallow | /*prefv4%3D |
| Disallow | /*subid%3D* |
| Disallow | /*pmax%3D* |
| Disallow | /*pmin%3D* |
| Disallow | /*start%3D* |
| Disallow | /*cgid%3D* |
| Disallow | /*sz%3D* |
| Disallow | /*format%3D* |
| Disallow | /*gclid%3D* |
| Disallow | /*linksrc%3D* |
| Disallow | /*variantID%3D* |
| Disallow | /Product-Variation* |
| Disallow | /SendToFriend-Start* |
| Disallow | /Product-GetBonusProducts* |
| Disallow | /COSamples-ExpressCheckout* |
| Disallow | /COUtils-RemoveGiftCertificate* |
| Disallow | /Wishlist-Add* |
| Disallow | /giftregistry-add* |
| Disallow | /wishlist-add* |
| Disallow | /Address* |
| Disallow | /VideoPlayer* |
| Disallow | /GiftRegistry* |
| Disallow | /AutoReplenishment* |
| Disallow | /Stores-ShowMap |
| Disallow | /CustomerService* |
| Disallow | /PaymentInstruments* |
| Disallow | /GiftCert* |
| Disallow | *PromoBanner-PromoLogin* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.aesop.com/sitemap_index.xml |