Log In Sign Up

jacobianengineering.com
robots.txt

Robots Exclusion Standard data for jacobianengineering.com

Resource Scan

Scan Details

Site Domain jacobianengineering.com
Base Domain jacobianengineering.com
Scan Status Ok
Last Scan2026-03-18T17:10:30+00:00
Next Scan 2026-04-17T17:10:30+00:00

Last Scan

Scanned2026-03-18T17:10:30+00:00
URL https://jacobianengineering.com/robots.txt
Domain IPs 104.26.10.175, 104.26.11.175, 172.67.71.9, 2606:4700:20::681a:aaf, 2606:4700:20::681a:baf, 2606:4700:20::ac43:4709
Response IP 104.26.11.175
Found Yes
Hash 2c3a5855b6eae8701da2b1e0694a31ad33e34dc952244dd18dbf3d105c90edd0
SimHash 8305d64366b4

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /login/
Disallow /dashboard/
Disallow /private/
Disallow /internal/
Disallow /*?
Disallow /staging/
Disallow /dev/
Disallow /test/
Disallow /form-handler/
Disallow /thank-you/
Disallow /confirmation/
Disallow /downloads/

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://jacobianengineering.com/sitemap.xml
sitemap https://jacobianengineering.com/sitemap-resources.xml
Back to top

Comments

  • Jacobian Engineering - robots.txt
  • Allow all crawlers by default
  • Block paths that don't contribute to SEO
  • Block query strings to prevent duplicate content
  • Block staging and test environments
  • Block form processing and confirmation pages
  • Block file downloads from indexing
  • Bing - throttle crawl rate (Google ignores Crawl-delay; use Search Console)
  • Block aggressive SEO crawlers
  • Sitemaps
Back to top