shop.harpersbazaar.com
robots.txt

Robots Exclusion Standard data for shop.harpersbazaar.com

Resource Scan

Scan Details

Site Domain shop.harpersbazaar.com
Base Domain harpersbazaar.com
Scan Status Ok
Last Scan2024-06-10T18:46:25+00:00
Next Scan 2024-07-10T18:46:25+00:00

Last Scan

Scanned2024-06-10T18:46:25+00:00
URL https://shop.harpersbazaar.com/robots.txt
Domain IPs 151.101.1.124, 151.101.129.124, 151.101.193.124, 151.101.65.124
Response IP 199.232.45.124
Found Yes
Hash c7ab0b045b638246d62e68726a29ff45d4df6c49afb7137494458aa85cec80b7
SimHash ed0d0881aa9b

Groups

*

Rule Path
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /stores/store/switch
Disallow /catalogsearch/
Disallow /productlist/
Disallow /newsletter/
Disallow /sendfriend/
Disallow /customize/
Disallow /checkout/
Disallow /wishlist/
Disallow /contacts/
Disallow /customer/
Disallow /review/
Disallow /ajax/
Disallow /poll/
Disallow /tag/
Disallow /marketplace/
Disallow /*/catalog/product_compare/
Disallow /*/catalog/category/view/
Disallow /*/catalog/product/view/
Disallow /*/stores/store/switch
Disallow /*/catalogsearch/
Disallow /*/productlist/
Disallow /*/newsletter/
Disallow /*/sendfriend/
Disallow /*/customize/
Disallow /*/checkout/
Disallow /*/wishlist/
Disallow /*/contacts/
Disallow /*/customer/
Disallow /*/review/
Disallow /*/ajax/
Disallow /*/poll/
Disallow /*/tag/
Disallow /*.php*
Disallow /*.sh*
Disallow /*?cat=*
Disallow /*?q=*
Disallow /*?query=*
Disallow /*.CVS
Disallow /*.Zip$
Disallow /*.Svn$
Disallow /*.Idea$
Disallow /*.Sql$
Disallow /*.Tgz$
Allow /*?p=*
Allow /*.js
Allow /*.css
Allow /*.jpg
Allow /*.jpeg
Allow /*.png
Allow /*.gif

Other Records

Field Value
sitemap https://shop.harpersbazaar.com/sitemap.xml

Comments

  • Crawlers Setup