insight.com
robots.txt

Robots Exclusion Standard data for insight.com

Resource Scan

Scan Details

Site Domain insight.com
Base Domain insight.com
Scan Status Ok
Last Scan2025-12-16T04:09:13+00:00
Next Scan 2026-01-15T04:09:13+00:00

Last Scan

Scanned2025-12-16T04:09:13+00:00
URL https://insight.com/robots.txt
Redirect https://www.insight.com/robots.txt
Redirect Domain www.insight.com
Redirect Base insight.com
Domain IPs 198.206.188.155
Redirect IPs 23.50.81.110
Response IP 104.69.158.211
Found Yes
Hash f73fa7ed889885ff36bd9f1a3684a3ff281351f648f6710a1cd3e2d6e490a65c
SimHash 0f389853ed53

Groups

*

Rule Path
Disallow /*?*
Allow /*.html
Disallow /en_US/search*.html
Disallow /insightweb/
Disallow /flytrap/
Disallow /content/dam/insight-web/*/solutions/service-provider/microsite/assets/
Disallow /content/dam/insight-web/*/pdfs/
Disallow /content/dam/insight/*/
Disallow /content/dam/global/*/pdfs/
Allow /insightweb/*.css$
Allow /*?qtype=
Allow /*?pq=
Allow /*?identifier=shopping
Allow /*?partnermessage

gptbot
chatgpt-user
google-extended
anthropic-ai
bingbot
googlebot
perplexitybot
youbot

Rule Path
Disallow

ccbot
facebookbot
neevaai

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.insight.com/sitemap.xml

Comments

  • Robots.txt for Insight.com
  • Default rules
  • Allowed AI crawlers
  • Blocked crawlers