Log In Sign Up

owltecheg.com
robots.txt

Robots Exclusion Standard data for owltecheg.com

Resource Scan

Scan Details

Site Domain owltecheg.com
Base Domain owltecheg.com
Scan Status Ok
Last Scan2025-09-05T05:54:11+00:00
Next Scan 2025-09-12T05:54:11+00:00

Last Scan

Scanned2025-09-05T05:54:11+00:00
URL https://owltecheg.com/robots.txt
Domain IPs 172.66.40.178, 172.66.43.78, 2606:4700:3108::ac42:28b2, 2606:4700:3108::ac42:2b4e
Response IP 172.66.40.178
Found Yes
Hash f9695469da5f70ab19c3366a153e0131f2b16a10282e06ab8b80733c395344fc
SimHash 0167d2d3e7a0

Groups

*

Rule Path
Disallow /admin/
Disallow /private/
Disallow /api/
Disallow /node_modules/
Allow /css/
Allow /js/
Allow /images/
Allow /assets/
Allow /Logo/
Allow /Portfolio/
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.gif$
Disallow /*.png$
Disallow /*.webp$
Allow /Logo/
Allow /Portfolio/
Allow /assets/images/
Disallow /.env
Disallow /.git/
Disallow /package.json
Disallow /package-lock.json
Disallow /vite.config.js
Disallow /tailwind.config.js
Disallow /postcss.config.js
Disallow /duplicate-content/
Disallow /mirror/
Allow /canonical/

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://owltecheg.com/sitemap.xml
Back to top

Comments

  • OwlTechEG - Robots.txt
  • Multi-domain configuration
  • https://owltecheg.com
  • https://www.owltecheg.com
  • https://owltecheg.net
  • https://www.owltecheg.net
  • Sitemap location (all domains redirect to owltecheg.com)
  • Crawl-delay for respectful crawling
  • General rules for all bots
  • Disallow admin or private areas (if any)
  • Allow important pages and assets
  • Block image hotlinking (but allow specific directories)
  • Allow specific image directories (overrides above rules)
  • Block access to development files
  • Additional rules for multi-domain setup
  • Prevent duplicate content issues
  • Allow canonical URLs
  • Specific rules for different bots (only customizations)
  • Block bad bots
Back to top

Warnings

  • 1 invalid line.
Back to top