ross-simons.com
robots.txt

Robots Exclusion Standard data for ross-simons.com

Resource Scan

Scan Details

Site Domain ross-simons.com
Base Domain ross-simons.com
Scan Status Ok
Last Scan2024-10-08T08:10:11+00:00
Next Scan 2024-11-07T08:10:11+00:00

Last Scan

Scanned2024-10-08T08:10:11+00:00
URL https://ross-simons.com/robots.txt
Redirect https://www.ross-simons.com/robots.txt
Redirect Domain www.ross-simons.com
Redirect Base ross-simons.com
Domain IPs 104.17.89.1, 104.17.90.1
Redirect IPs 104.17.89.1, 104.17.90.1
Response IP 104.17.90.1
Found Yes
Hash 81a491ec5cf91f8245252c7450a218c9451a08b66db117879bb5ecb542d30e74
SimHash 63449c56ce2e

Groups

*

Rule Path
Disallow /*prefn3%3D*
Disallow /*prefv3%3D*
Disallow /*prefn4%3D*
Disallow /*prefv4%3D*
Disallow /*prefn5%3D*
Disallow /*prefv5%3D*
Disallow /*prefn6%3D*
Disallow /*prefv6%3D*
Disallow /*prefn7%3D*
Disallow /*prefv7%3D*
Disallow /*pmin%3D*
Disallow /*pmax%3D*
Disallow *%7C*
Disallow *%7C*
Disallow /cart*
Disallow /checkout*
Disallow */demandware.store*
Allow */demandware.store/*/Search-UpdateGrid
Allow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

facebookbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

facebookexternalhit/1.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

facebookexternalhit/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://www.ross-simons.com/sitemap_index.xml

Comments

  • Prevent indexing of raw search refinements, price ranges, sorting rules
  • Disallow crawling of 3+ Refinements
  • Prevent indexing of checkout and cart sessions
  • Prevent indexing of un-aliased storefront URLs
  • Copy and uncomment to allow access to certain resources
  • Allow: */demandware.store/*/Pipeline-Name
  • Taxonomy Sitemaps
  • Crawl delays for common bots