aubert.com
robots.txt

Robots Exclusion Standard data for aubert.com

Resource Scan

Scan Details

Site Domain aubert.com
Base Domain aubert.com
Scan Status Ok
Last Scan2025-11-01T00:09:35+00:00
Next Scan 2025-12-01T00:09:35+00:00

Last Scan

Scanned2025-11-01T00:09:35+00:00
URL https://aubert.com/robots.txt
Redirect https://www.aubert.com/robots.txt
Redirect Domain www.aubert.com
Redirect Base aubert.com
Domain IPs 104.20.31.183, 172.66.173.60, 2606:4700:10::6814:1fb7, 2606:4700:10::ac42:ad3c
Redirect IPs 104.20.31.183, 172.66.173.60, 2606:4700:10::6814:1fb7, 2606:4700:10::ac42:ad3c
Response IP 172.66.173.60
Found Yes
Hash 6650cd807ba9dbbe6f5abf8bdd9147b080b836061f4f5df8d4104ed9fb7eeca9
SimHash 6a52c732ce77

Groups

*

Rule Path
Disallow /my-account
Disallow /checkout
Disallow /*search/
Disallow /login
Disallow /cart
Disallow /magasin/*/contact
Disallow /naissance/magasin
Disallow /*/setmystore
Disallow /*/quickView
Disallow /*/naissance
Disallow */c/favoris
Disallow /*/review
Disallow /*00000000/produits
Disallow /conseils/?s=
Disallow /*sfterm%3D
Disallow /*show%3D
Disallow /*text%3D
Disallow /*sort%3D
Disallow /*qh%3D
Disallow /*q%3D
Disallow /*/cat-sans-promo/
Disallow /*/cat-sans-promo-ch/
Disallow /*/cat-sans-promo-be/
Disallow /*/sea-costly/
Disallow /*/sea-flukes/
Disallow /*/sea-meh/
Disallow /*/sea-profitable/
Disallow /*/sea-zeroconv/
Disallow /*/sea-zombie/
Disallow /load/analytics/tracking
Disallow /*bundleVariation%3D
Disallow /*catalogParam*topicId*
Disallow /*emsParam*email*
Disallow /*?exp*p*
Disallow /*%26exp*p*
Disallow /*postalCode%3D
Disallow /*title%3D
Disallow /*tog_sb%3D
Disallow /*toSubid%3D
Disallow /*xtor%3D
Disallow *page%3D5*
Allow *page%3D5$
Disallow *page%3D6*
Allow *page%3D6$
Disallow *page%3D7*
Allow *page%3D7$
Disallow *page%3D8*
Allow *page%3D8$
Disallow *page%3D9*
Allow *page%3D9$
Disallow *page%3D1*
Allow *page%3D1$
Allow *page%3D10$
Allow *page%3D11$
Allow *page%3D12$
Allow *page%3D13$
Allow *page%3D14$
Allow *page%3D15$
Allow *page%3D16$
Allow *page%3D17$
Allow *page%3D18$
Allow *page%3D19$
Disallow *page%3D2*
Allow *page%3D2$
Allow *page%3D20$
Allow *page%3D21$
Allow *page%3D22$
Allow *page%3D23$
Allow *page%3D24$
Allow *page%3D25$
Allow *page%3D26$
Allow *page%3D27$
Allow *page%3D28$
Allow *page%3D29$

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.aubert.com/sitemap.xml
sitemap https://www.aubert.com/conseils/sitemap_index.xml
sitemap https://magasins.aubert.com/sitemap_index.xml

Comments

  • Block access to specific groups of pages
  • Block SEA + sans promo sub-folders
  • Block /load/analytics
  • Block parameters URLs
  • Block paginated > 50
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot