Log In Sign Up

triad-eng.com
robots.txt

Robots Exclusion Standard data for triad-eng.com

Resource Scan

Scan Details

Site Domain triad-eng.com
Base Domain triad-eng.com
Scan Status Ok
Last Scan2025-10-24T19:00:53+00:00
Next Scan 2025-11-23T19:00:53+00:00

Last Scan

Scanned2025-10-24T19:00:53+00:00
URL https://triad-eng.com/robots.txt
Domain IPs 34.174.63.173
Response IP 34.174.63.173
Found Yes
Hash bc5cb73fb17a900f0dd637d30e4f92f8e17dfdc9d6f82f3cab51d8d4607c0ea5
SimHash 00109f40eca3

Groups

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

applebot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

perplexity-user

Rule Path
Allow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

grok

Rule Path
Disallow /

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Allow /
Disallow /wp-admin/
Disallow /login/
Disallow /wp-login.php
Disallow /search/
Disallow /*?s=*
Disallow /tag/
Disallow /category/
Disallow /feed/
Allow /wp-admin/admin-ajax.php
Back to top

Other Records

Field Value
sitemap https://www.triad-eng.com/sitemap_index.xml
Back to top

Comments

  • TRIAD Engineering — Robots Policy: Balanced (no AI training)
  • Classic search engines
  • Allow on-demand assistants (browsers fetching per user request)
  • Block AI/model-training crawlers
  • Manage crawl rate for aggressive bots
  • Default rules for all crawlers
  • Sitemap
Back to top