Log In Sign Up

polymash.com
robots.txt

Robots Exclusion Standard data for polymash.com

Resource Scan

Scan Details

Site Domain polymash.com
Base Domain polymash.com
Scan Status Ok
Last Scan2025-10-17T02:36:41+00:00
Next Scan 2025-11-16T02:36:41+00:00

Last Scan

Scanned2025-10-17T02:36:41+00:00
URL https://polymash.com/robots.txt
Domain IPs 35.215.124.137
Response IP 35.215.124.137
Found Yes
Hash c471629f1d47827a41a461f14deb9a3722ec5a8cf3505e064db12339638a8edb
SimHash 483e9842cdd4

Groups

googlebot

Rule Path
Allow /

gptbot

Product Comment
gptbot OpenAI
Rule Path
Allow /

Other Records

Field Value Comment
crawl-delay 10 Reduce simultaneous pulls without blocking[5][11]

claudebot

Product Comment
claudebot Anthropic
Rule Path
Allow /

Other Records

Field Value Comment
crawl-delay 15 Claimed to respect robots.txt

twitterbot

Rule Path Comment
Allow / Keeps Twitter card previews working

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/

Other Records

Field Value Comment
crawl-delay 8 Slows all other unidentified crawlers

semrushbot

Rule Path
Disallow /

semrushbot-desktop

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

google-extended

Product Comment
google-extended Google’s opt-out for AI training
Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://polymash.com/sitemap_index.xml
Back to top

Comments

  • ---------- FRIENDLY / ESSENTIAL BOTS ----------
  • ---------- GLOBAL DEFAULT ----------
  • ---------- AGGRESSIVE SEO / AI SCRAPERS ----------
  • SEMrush – multiple variants published by Semrush[4][13]
  • Ahrefs crawlers (marketing + Site Audit)[6][15]
  • AI training collectors that create heavy load[8][17]
  • ---------- SITEMAPS ----------
Back to top

Warnings

  • 1 invalid line.
Back to top