spectra-physics.com
robots.txt

Robots Exclusion Standard data for spectra-physics.com

Resource Scan

Scan Details

Site Domain spectra-physics.com
Base Domain spectra-physics.com
Scan Status Ok
Last Scan2025-09-16T04:42:27+00:00
Next Scan 2025-10-16T04:42:27+00:00

Last Scan

Scanned2025-09-16T04:42:27+00:00
URL https://www.spectra-physics.com/robots.txt
Redirect https://www.spectra-physics.com/en/robots.txt
Domain IPs 138.113.144.11
Response IP 138.113.144.11
Found Yes
Hash bbda76e1c7a0b1b96da7cbc1c62d833afc0296a8551d2b3481168ab00d02e548
SimHash 3c5047d6eff2

Groups

*

Rule Path
Disallow /en/cart
Disallow /en/checkout
Disallow /en/my-account

Other Records

Field Value Comment
crawl-delay 10 10 seconds between page requests

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.spectra-physics.com/medias/sys_master/root/h3d/hc9/10045391896606/Homepage-en-USD-2110483451888372487/Homepage-en-USD-2110483451888372487.xml
sitemap https://www.spectra-physics.com/medias/sys_master/root/h4f/hcc/10045391962142/Product-en-USD-9110919333846104216/Product-en-USD-9110919333846104216.xml
sitemap https://www.spectra-physics.com/medias/sys_master/root/h07/he9/10045392027678/CategoryLanding-en-USD-16107367929076265157/CategoryLanding-en-USD-16107367929076265157.xml
sitemap https://www.spectra-physics.com/medias/sys_master/root/h45/he8/10045392093214/Category-en-USD-7924371575081904875/Category-en-USD-7924371575081904875.xml
sitemap https://www.spectra-physics.com/medias/sys_master/root/h36/he2/10045392224286/Content-en-USD-3606378132880723484/Content-en-USD-3606378132880723484.xml

Comments

  • For all robots
  • Block access to specific groups of pages
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot