Log In Sign Up

akhurst.com
robots.txt

Robots Exclusion Standard data for akhurst.com

Resource Scan

Scan Details

Site Domain akhurst.com
Base Domain akhurst.com
Scan Status Ok
Last Scan2026-01-27T16:32:20+00:00
Next Scan 2026-02-10T16:32:20+00:00

Last Scan

Scanned2026-01-27T16:32:20+00:00
URL https://akhurst.com/robots.txt
Domain IPs 104.26.0.201, 104.26.1.201, 172.67.73.164, 2606:4700:20::681a:1c9, 2606:4700:20::681a:c9, 2606:4700:20::ac43:49a4
Response IP 172.67.73.164
Found Yes
Hash f123bef515d61643ddce17719729881f8357bd15e149fe551c2651c04534a549
SimHash 6039ca02e693

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /cgi-bin/
Disallow /cart/
Disallow /checkout/
Disallow /?s=
Allow /wp-admin/admin-ajax.php

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://cantekamerica.com/sitemap_index.xml
Back to top

Comments

  • /public_html/robots.txt (Used by Cantek)
  • General rules for all bots
  • Explicit allow for major search engines
  • Block known aggressive or non-beneficial bots
  • Crawl-delay: 10
  • Sitemap location
Back to top