Log In Sign Up

builtrobotics.com
robots.txt

Robots Exclusion Standard data for builtrobotics.com

Resource Scan

Scan Details

Site Domain builtrobotics.com
Base Domain builtrobotics.com
Scan Status Ok
Last Scan2026-02-12T07:30:30+00:00
Next Scan 2026-02-19T07:30:30+00:00

Last Scan

Scanned2026-02-12T07:30:30+00:00
URL https://builtrobotics.com/robots.txt
Redirect https://www.builtrobotics.com/robots.txt
Redirect Domain www.builtrobotics.com
Redirect Base builtrobotics.com
Domain IPs 172.66.40.104, 172.66.43.152, 2606:4700:3108::ac42:2868, 2606:4700:3108::ac42:2b98
Redirect IPs 172.66.40.104, 172.66.43.152, 2606:4700:3108::ac42:2868, 2606:4700:3108::ac42:2b98
Response IP 172.66.43.152
Found Yes
Hash f65b3c637712f28533f3e9cf85c35bcf83c930ad308a2a8bb9dd81347b100a00
SimHash 623010522d90

Groups

*

Rule Path
Disallow /cpresources/
Disallow /vendor/
Disallow /.env
Disallow /cache/
Disallow /*?*

oai-searchbot
chatgpt-user
perplexitybot
firecrawlagent
andibot
exabot
phindbot
youbot

Rule Path
Allow /

gptbot
ccbot
google-extended

Rule Path
Disallow /

googlebot
bingbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.builtrobotics.com/sitemaps-1-sitemap.xml
sitemap https://built.solar/sitemaps-1-sitemap.xml
Back to top

Comments

  • robots.txt for https://www.builtrobotics.com/
  • live - don't allow web crawlers to index cpresources/ or vendor/
  • block all URLs with a query string to avoid duplicate content
  • Allow AI search and agent use
  • Disallow AI training data collection
  • Allow traditional search indexing
Back to top