Log In Sign Up

haynes.com
robots.txt

Robots Exclusion Standard data for haynes.com

Resource Scan

Scan Details

Site Domain haynes.com
Base Domain haynes.com
Scan Status Ok
Last Scan2024-05-11T21:46:21+00:00
Next Scan 2024-06-10T21:46:21+00:00

Last Scan

Scanned2024-05-11T21:46:21+00:00
URL https://haynes.com/robots.txt
Domain IPs 52.84.229.103, 52.84.229.29, 52.84.229.46, 52.84.229.97
Response IP 52.84.229.46
Found Yes
Hash 6dd98cb023f7a1452f2218415b0d3195277e3e4c5504fc1fb80a37e7a26736c8
SimHash 266c50476e62

Groups

*

Rule Path
Disallow /global-catalog/
Disallow /en-us/includes/
Disallow /en-us/misc/
Disallow /en-us/modules/
Disallow /en-us/profiles/
Disallow /en-us/scripts/
Disallow /en-us/themes/
Disallow /en-us/CHANGELOG.txt
Disallow /en-us/cron.php
Disallow /en-us/INSTALL.mysql.txt
Disallow /en-us/INSTALL.pgsql.txt
Disallow /en-us/INSTALL.sqlite.txt
Disallow /en-us/install.php
Disallow /en-us/INSTALL.txt
Disallow /en-us/LICENSE.txt
Disallow /en-us/MAINTAINERS.txt
Disallow /en-us/update.php
Disallow /en-us/UPGRADE.txt
Disallow /en-us/xmlrpc.php
Disallow /en-us/admin/
Disallow /en-us/comment/reply/
Disallow /en-us/filter/tips/
Disallow /en-us/node/add/
Disallow /en-us/user/register/
Disallow /en-us/user/password/
Disallow /en-us/user/logout/
Disallow /en-us/search
Disallow /en-us/?q=admin%2F
Disallow /en-us/?q=comment%2Freply%2F
Disallow /en-us/?q=filter%2Ftips%2F
Disallow /en-us/?q=node%2Fadd%2F
Disallow /en-us/?q=search%2F
Disallow /en-us/?q=user%2Fpassword%2F
Disallow /en-us/?q=user%2Fregister%2F
Disallow /en-us/?q=user%2Flogin%2F
Disallow /en-us/?q=user%2Flogout%2F
Disallow /en-us/catalog/add-to-cart
Disallow /en-us/?q=catalog%2Fadd-to-cart%2F
Disallow /en-us/cas*
Disallow /en-us/es/account?
Disallow /en-us/es/cas?
Disallow /en-us/es/
Disallow /en-us/ajax/
Disallow /en-us/cart
Disallow /en-us/cart/*
Disallow /en-us/checkout
Disallow /en-us/checkout/*
Disallow /en-gb/includes/
Disallow /en-gb/misc/
Disallow /en-gb/modules/
Disallow /en-gb/profiles/
Disallow /en-gb/scripts/
Disallow /en-gb/themes/
Disallow /en-gb/CHANGELOG.txt
Disallow /en-gb/cron.php
Disallow /en-gb/INSTALL.mysql.txt
Disallow /en-gb/INSTALL.pgsql.txt
Disallow /en-gb/INSTALL.sqlite.txt
Disallow /en-gb/install.php
Disallow /en-gb/INSTALL.txt
Disallow /en-gb/LICENSE.txt
Disallow /en-gb/MAINTAINERS.txt
Disallow /en-gb/update.php
Disallow /en-gb/UPGRADE.txt
Disallow /en-gb/xmlrpc.php
Disallow /en-gb/admin/
Disallow /en-gb/comment/reply/
Disallow /en-gb/filter/tips/
Disallow /en-gb/node/add/
Disallow /en-gb/user/register/
Disallow /en-gb/user/password/
Disallow /en-gb/user/logout/
Disallow /en-gb/search
Disallow /en-gb/?q=admin%2F
Disallow /en-gb/?q=comment%2Freply%2F
Disallow /en-gb/?q=filter%2Ftips%2F
Disallow /en-gb/?q=node%2Fadd%2F
Disallow /en-gb/?q=search%2F
Disallow /en-gb/?q=user%2Fpassword%2F
Disallow /en-gb/?q=user%2Fregister%2F
Disallow /en-gb/?q=user%2Flogin%2F
Disallow /en-gb/?q=user%2Flogout%2F
Disallow /en-gb/catalog/add-to-cart
Disallow /en-gb/?q=catalog%2Fadd-to-cart%2F
Disallow /en-gb/cas*
Disallow /en-gb/es/account?
Disallow /en-gb/es/cas?
Disallow /en-gb/es/
Disallow /en-gb/ajax/
Disallow /en-gb/cart
Disallow /en-gb/cart/*
Disallow /en-gb/checkout
Disallow /en-gb/checkout/*
Disallow /en-au/includes/
Disallow /en-au/misc/
Disallow /en-au/modules/
Disallow /en-au/profiles/
Disallow /en-au/scripts/
Disallow /en-au/themes/
Disallow /en-au/CHANGELOG.txt
Disallow /en-au/cron.php
Disallow /en-au/INSTALL.mysql.txt
Disallow /en-au/INSTALL.pgsql.txt
Disallow /en-au/INSTALL.sqlite.txt
Disallow /en-au/install.php
Disallow /en-au/INSTALL.txt
Disallow /en-au/LICENSE.txt
Disallow /en-au/MAINTAINERS.txt
Disallow /en-au/update.php
Disallow /en-au/UPGRADE.txt
Disallow /en-au/xmlrpc.php
Disallow /en-au/admin/
Disallow /en-au/comment/reply/
Disallow /en-au/filter/tips/
Disallow /en-au/node/add/
Disallow /en-au/user/register/
Disallow /en-au/user/password/
Disallow /en-au/user/logout/
Disallow /en-au/search
Disallow /en-au/?q=admin%2F
Disallow /en-au/?q=comment%2Freply%2F
Disallow /en-au/?q=filter%2Ftips%2F
Disallow /en-au/?q=node%2Fadd%2F
Disallow /en-au/?q=search%2F
Disallow /en-au/?q=user%2Fpassword%2F
Disallow /en-au/?q=user%2Fregister%2F
Disallow /en-au/?q=user%2Flogin%2F
Disallow /en-au/?q=user%2Flogout%2F
Disallow /en-au/catalog/add-to-cart
Disallow /en-au/?q=catalog%2Fadd-to-cart%2F
Disallow /en-au/cas*
Disallow /en-au/es/account?
Disallow /en-au/es/cas?
Disallow /en-au/es/
Disallow /en-au/ajax/
Disallow /en-au/cart
Disallow /en-au/cart/*
Disallow /en-au/checkout
Disallow /en-au/checkout/*
Disallow /en-gb/microsites/*

Other Records

Field Value
crawl-delay 10

gptbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://haynes.com/en-us/sitemap.xml
sitemap https://haynes.com/en-gb/sitemap.xml
sitemap https://haynes.com/en-au/sitemap.xml
Back to top

Comments

  • Global Catalog
  • robots_include.txt
  • Note: this robots.txt file will be aggregrated by the
  • root robots.txt script as having multiple separate robots.txt files
  • is detrimental to SEO.
  • E.g. https://haynes.com/robots.txt (root redirect script)
  • --> loads https://haynes.com/en-gb/robots_include.txt
  • --> loads https://haynes.com/en-us/robots_include.txt
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Custom paths
  • Checkout paths
  • Sitemaps
  • robots_include.txt
  • Note: this robots.txt file will be aggregrated by the
  • root robots.txt script as having multiple separate robots.txt files
  • is detrimental to SEO.
  • E.g. https://haynes.com/robots.txt (root redirect script)
  • --> loads https://haynes.com/en-gb/robots_include.txt
  • --> loads https://haynes.com/en-us/robots_include.txt
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Custom paths
  • Checkout paths
  • Sitemaps
  • robots_include.txt
  • Note: this robots.txt file will be aggregrated by the
  • root robots.txt script as having multiple separate robots.txt files
  • is detrimental to SEO.
  • E.g. https://haynes.com/robots.txt (root redirect script)
  • --> loads https://haynes.com/en-gb/robots_include.txt
  • --> loads https://haynes.com/en-us/robots_include.txt
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Custom paths
  • Checkout paths
  • Sitemaps
Back to top