Log In Sign Up

harborfreight.com
robots.txt

Robots Exclusion Standard data for harborfreight.com

Resource Scan

Scan Details

Site Domain harborfreight.com
Base Domain harborfreight.com
Scan Status Ok
Last Scan2024-05-19T11:48:33+00:00
Next Scan 2024-06-18T11:48:33+00:00

Last Scan

Scanned2024-05-19T11:48:33+00:00
URL https://harborfreight.com/robots.txt
Redirect https://www.harborfreight.com/robots.txt
Redirect Domain www.harborfreight.com
Redirect Base harborfreight.com
Domain IPs 151.101.1.124, 151.101.129.124, 151.101.193.124, 151.101.65.124
Redirect IPs 151.101.1.124, 151.101.129.124, 151.101.193.124, 151.101.65.124
Response IP 199.232.45.124
Found Yes
Hash be73a5f6df22a15331faf166b0850f3440a4c63e305fd9d9f2c55d1de0ea187b
SimHash 390addb366d8

Groups

*

Rule Path
Disallow /catalogsearch/
Disallow /test/
Disallow /customer/
Disallow /checkout/
Disallow /catalog/
Disallow /media/
Disallow /merchandising-promotions/
Disallow /lib/
Disallow /*.php$
Disallow /pkginfo/
Disallow /report/
Disallow /var/
Disallow /catalog/
Disallow /customer/%27
Disallow /sendfriend/%27
Disallow /review/%27
Disallow /*SID%3D

googlebot

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

yandex

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

duckduckbot

Rule Path
Allow /

facebookbot

Rule Path
Allow /

twitterbot

Rule Path
Allow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.harborfreight.com/sitemap.xml
Back to top

Comments

  • Harbor Freight Tools Robots.txt File
  • _ _ _ ______ _ _ _ _____ _
  • | | | | | | | ___| (_) | | | | |_ _| | |
  • | |_| | __ _ _ __ | |__ ___ _ __ | |_ _ __ ___ _ __ _ | |__ | |_ | | ___ ___ | | ___
  • | _ | / _` || '__|| '_ \ / _ \ | '__| | _| | '__| / _ \| | / _` || '_ \ | __| | | / _ \ / _ \ | |/ __|
  • | | | || (_| || | | |_) || (_) || | | | | | | __/| || (_| || | | || |_ | | | (_) || (_) || |\__ \
  • \_| |_/ \__,_||_| |_.__/ \___/ |_| \_| |_| \___||_| \__, ||_| |_| \__| \_/ \___/ \___/ |_||___/
  • __/ |
  • |___/
  • Welcome! Glad you found us!
  • Wait, you're not a robot. But I guess humans fine.
  • I'm just making sure no disallowed subfolders get through.
  • Here's a joke for you humans. How do robots eat salsa?.... With micro-chips!
  • Looking for a better use of your time? Check out: https://jobs.harborfreight.com/
  • Directories & URLs updated 1/05/2023
  • Searchbots updated 1/05/2023
  • Sitemap updated 1/05/2023
Back to top