Log In Sign Up

jeng.us
robots.txt

Robots Exclusion Standard data for jeng.us

Resource Scan

Scan Details

Site Domain jeng.us
Base Domain jeng.us
Scan Status Ok
Last Scan2026-03-08T01:01:08+00:00
Next Scan 2026-04-07T01:01:08+00:00

Last Scan

Scanned2026-03-08T01:01:08+00:00
URL http://jeng.us/robots.txt
Redirect https://jacobianengineering.com/robots.txt
Redirect Domain jacobianengineering.com
Redirect Base jacobianengineering.com
Domain IPs 16.15.186.174, 16.15.223.190, 16.182.32.237, 52.216.211.29, 52.217.192.117, 52.217.201.197, 54.231.129.133
Redirect IPs 104.26.10.175, 104.26.11.175, 172.67.71.9, 2606:4700:20::681a:aaf, 2606:4700:20::681a:baf, 2606:4700:20::ac43:4709
Response IP 172.67.71.9
Found Yes
Hash f6a18e8e13629c75ca811de112fd5dd4e5645734df15583eda72f68c487261c5
SimHash 8305d64366b4

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /login/
Disallow /dashboard/
Disallow /private/
Disallow /internal/
Disallow /*?
Disallow /staging/
Disallow /dev/
Disallow /test/
Disallow /form-handler/
Disallow /thank-you/
Disallow /confirmation/
Disallow /downloads/

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://jacobianengineering.com/sitemap.xml
sitemap https://jacobianengineering.com/sitemap-resources.xml
Back to top

Comments

  • Jacobian Engineering - robots.txt
  • Allow all crawlers by default
  • Block paths that don't contribute to SEO
  • Block query strings to prevent duplicate content
  • Block staging and test environments
  • Block form processing and confirmation pages
  • Block file downloads from indexing
  • Bing - throttle crawl rate (Google ignores Crawl-delay; use Search Console)
  • Block aggressive SEO crawlers
  • Sitemaps
Back to top