Log In Sign Up

crystaldreamsworld.com
robots.txt

Robots Exclusion Standard data for crystaldreamsworld.com

Resource Scan

Scan Details

Site Domain crystaldreamsworld.com
Base Domain crystaldreamsworld.com
Scan Status Ok
Last Scan2025-10-09T21:15:49+00:00
Next Scan 2025-11-08T21:15:49+00:00

Last Scan

Scanned2025-10-09T21:15:49+00:00
URL https://crystaldreamsworld.com/robots.txt
Domain IPs 104.21.59.23, 172.67.211.204, 2606:4700:3030::6815:3b17, 2606:4700:3030::ac43:d3cc
Response IP 104.21.59.23
Found Yes
Hash def62e48fb104feeb160b8c3d5f0b5d3fac518c09ea5fe1cd60a2f5227de53be
SimHash 3113fa10449e

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*
Allow /wp-admin/admin-ajax.php

chatgpt-user

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

perplexitybot

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

xai-crawler

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

anthropicbot

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

claude-web

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

claudebot

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

oai-searchbot

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

google-extended

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

bytespider

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

applebot-extended

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

applebot

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

msnbot

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

meta-externalagent

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

amazonbot

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

facebookbot

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

duckassistbot

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

googlebot-image

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

adsbot-google

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*

googlebot-video

Rule Path
Allow */product*
Allow */product-category*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow *add_to_wishlist%3D*
Disallow */?add-to-cart=*
Disallow */?product_view=*
Disallow */?attribute_*
Disallow *?orderby=*
Disallow *?fb_comment_id=*
Disallow *?_escaped_fragment_=*
Disallow */social/*
Disallow */?s=*
Disallow */?wc-ajax=*
Disallow *%26product_view%3D*
Back to top

Other Records

Field Value
sitemap https://crystaldreamsworld.com/sitemap.xml
sitemap https://crystaldreamsworld.com/post-sitemap.xml
sitemap https://crystaldreamsworld.com/page-sitemap.xml
sitemap https://crystaldreamsworld.com/product-sitemap.xml
sitemap https://crystaldreamsworld.com/product-sitemap2.xml
sitemap https://crystaldreamsworld.com/product-sitemap3.xml
sitemap https://crystaldreamsworld.com/product-sitemap4.xml
sitemap https://crystaldreamsworld.com/product-sitemap5.xml
sitemap https://crystaldreamsworld.com/product-sitemap6.xml
sitemap https://crystaldreamsworld.com/product_cat-sitemap.xml
Back to top

Comments

  • ------------------------------------------------------------------------------------------------
  • Autoriser les Bots IA à crawler les pages produits, catégories mais pas les paramètres d'url déjà disallow plus haut.
  • ------------------------------------------------------------------------------------------------
Back to top