heruniverse.com
robots.txt

Robots Exclusion Standard data for heruniverse.com

Resource Scan

Scan Details

Site Domain heruniverse.com
Base Domain heruniverse.com
Scan Status Ok
Last Scan2025-10-25T14:29:54+00:00
Next Scan 2025-11-24T14:29:54+00:00

Last Scan

Scanned2025-10-25T14:29:54+00:00
URL https://heruniverse.com/robots.txt
Redirect https://www.heruniverse.com/robots.txt
Redirect Domain www.heruniverse.com
Redirect Base heruniverse.com
Domain IPs 104.16.198.245, 104.16.199.245
Redirect IPs 104.16.198.245, 104.16.199.245
Response IP 104.16.199.245
Found Yes
Hash 136146441048f93265d9c156682abc7abc0aa4c2312ec17bb00cd74a825eec54
SimHash 203456744c1e

Groups

*

Rule Path
Disallow /OOBO/
Disallow /mkt/
Disallow /*pmax%3D*
Disallow /*prefn1%3D*
Disallow /*prefn2%3D*
Disallow /*prefn3%3D*
Disallow /*prefn4%3D*
Disallow /*prefn1%3D*
Disallow /*prefv2%3D*
Disallow /*prefv3%3D*
Disallow /*prefv4%3D*
Disallow /*srule%3D*
Disallow /*sz%3D*start%3D*
Disallow /*start%3D*sz%3D*
Disallow /*queryBRT%3D*
Disallow /*?q=*
Disallow /*%2B*%2B*-*$
Allow /*theme?q=*
Allow /tees-2/?q=*
Disallow */in-store-deals*
Disallow /*dwcont%3D*
Disallow /*pid%3D*
Disallow /*bvstate*
Disallow /*bmUID%3D*
Disallow /account*
Disallow /cart*
Disallow /checkout*
Disallow /order-history*
Disallow /search*
Disallow /wishlist*
Disallow /*/favicon.ico
Allow /favicon.ico
Disallow /srf.*
Allow /shop-by-license/*/srf.*
Disallow /srf.*_*
Disallow */demandware.store*
Disallow */demandware.store/*/Search-UpdateGrid

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

facebookbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

facebookexternalhit/1.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

facebookexternalhit/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://www.heruniverse.com/sitemap_index.xml

Comments

  • Disallow Nodes with price ranges or multiple refinement values
  • Prevent indexing of common SFCC variables
  • Prevent indexing of multiple pages of Bazaarvoice reviews
  • Disallow access to cart/checkout
  • Relative URL favicon.ico issue
  • Search Refinement URLs
  • Crawl delays for common bots