Log In Sign Up

kemet.com
robots.txt

Robots Exclusion Standard data for kemet.com

Resource Scan

Scan Details

Site Domain kemet.com
Base Domain kemet.com
Scan Status Ok
Last Scan2024-10-26T10:10:15+00:00
Next Scan 2024-11-25T10:10:15+00:00

Last Scan

Scanned2024-10-26T10:10:15+00:00
URL https://www.kemet.com/robots.txt
Domain IPs 52.254.109.59
Response IP 52.254.109.59
Found Yes
Hash 0e84f2137cf2ff0ef8159cb12bb89ff9ab8b1dcb6b0a617a9b2873d2a4872ede
SimHash b8c0bf9eede4

Groups

*

Rule Path
Disallow /us/en/cart

Other Records

Field Value Comment
crawl-delay 10 10 seconds between page requests

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

*

Rule Path
Disallow /content/kemet/master

*

Rule Path
Disallow /content/nexensos/nexensos-blueprint
Back to top

Other Records

Field Value
sitemap https://www.kemet.com/sitemap.xml
Back to top

Comments

  • For all robots
  • Block access to specific groups of pages
  • Allow search crawlers to discover the sitemap
  • This sitemap will include the other languages as well.
  • There is no need for separate sitemaps.
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Block SEMrushBot
  • Block SEMrushBot from crawling your site for different SEO and technical issues:
  • Block SEMrushBot from crawling your site for Backlink Audit tool:
  • Block SEMrushBot from crawling your site for On Page SEO Checker tool and similar tools:
  • Block SEMrushBot from checking URLs your site for SWA tool:
  • Block SEMrushBot from crawling your site for Content Analyzer and Post Tracking tools:
  • Block SEMrushBot from crawling your site for Brand Monitoring:
  • Block Baiduspider
  • Block Yandex
  • Block Kemet Master
  • Block Nexensos Blueprint Sites
Back to top

Warnings

  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top