Log In Sign Up

gates.com
robots.txt

Robots Exclusion Standard data for gates.com

Resource Scan

Scan Details

Site Domain gates.com
Base Domain gates.com
Scan Status Ok
Last Scan2024-10-29T08:07:53+00:00
Next Scan 2024-11-28T08:07:53+00:00

Last Scan

Scanned2024-10-29T08:07:53+00:00
URL https://gates.com/robots.txt
Redirect https://www.gates.com/robots.txt
Redirect Domain www.gates.com
Redirect Base gates.com
Domain IPs 104.18.37.236, 172.64.150.20, 2606:4700:4400::6812:25ec, 2606:4700:4400::ac40:9614
Redirect IPs 104.18.37.236, 172.64.150.20, 2606:4700:4400::6812:25ec, 2606:4700:4400::ac40:9614
Response IP 172.64.150.20
Found Yes
Hash 1a1ce7887c8cb51f5d708162647fe9deb068fa8bdbb6e7c3207edc41ebc456c0
SimHash 38a6f71f65e4

Groups

*

Rule Path
Disallow /us/en/cart
Disallow /us/en/checkout
Disallow /us/en/my-account
Disallow /index.php/
Disallow /app/
Disallow /lib/
Disallow /*?
Disallow /*.php%24%7Bcolor%7D
Disallow /pkginfo/
Disallow /report/
Disallow /var/
Disallow /catalog/
Disallow /customer/
Disallow /sendfriend/
Disallow /review/
Disallow /*SID%3D
Disallow /cn/en
Disallow /cn/zh
Disallow /br/en
Disallow /br/pt
Disallow /th/en
Disallow /vn/en
Disallow /search*
Disallow /*/*/search*
Disallow /*/*/*/*/search*
Disallow /us/en/ymm/test-pages-section
Disallow /us/en/test

Other Records

Field Value Comment
crawl-delay 10 10 seconds between page requests

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.gates.com/us/en/sitemap.xml
Back to top

Comments

  • For all robots
  • Block access to specific groups of pages
  • Allow search crawlers to discover the sitemap
  • This sitemap will include the other languages as well.
  • There is no need for separate sitemaps.
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Block SEMrushBot
  • Block SEMrushBot from crawling your site for different SEO and technical issues:
  • Block SEMrushBot from crawling your site for Backlink Audit tool:
  • Block SEMrushBot from crawling your site for On Page SEO Checker tool and similar tools:
  • Block SEMrushBot from checking URLs your site for SWA tool:
  • Block SEMrushBot from crawling your site for Content Analyzer and Post Tracking tools:
  • Block SEMrushBot from crawling your site for Brand Monitoring:
  • Block Baiduspider
Back to top

Warnings

  • `noindex` is not a known field.
  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top