capitalgroup.com
robots.txt

Robots Exclusion Standard data for capitalgroup.com

Resource Scan

Scan Details

Site Domain capitalgroup.com
Base Domain capitalgroup.com
Scan Status Ok
Last Scan2024-05-22T08:25:24+00:00
Next Scan 2024-06-05T08:25:24+00:00

Last Scan

Scanned2024-05-22T08:25:24+00:00
URL https://capitalgroup.com/robots.txt
Redirect https://www.capitalgroup.com:443/robots.txt
Redirect Domain www.capitalgroup.com
Redirect Base capitalgroup.com
Domain IPs 15.197.203.180, 3.33.191.209
Redirect IPs 23.15.110.253
Response IP 23.15.110.253
Found Yes
Hash 494dafc123e9d3ce5c5c9dc58744cfe8233bca1c5127e4914ee147a0d66914b9
SimHash 31301357c9e0

Groups

atomz/1.0

Rule Path
Disallow

ia_archiver

Rule Path
Disallow /

visweb

Rule Path
Disallow /

*

Rule Path
Disallow /*/error
Disallow /*/preferences
Allow /advisor/preferences/communication-preferences.htm
Disallow /*/accounts
Allow /*/accounts/login.htm
Disallow /*/pardon
Disallow /*/search
Disallow /advisor/merrill-lynch.html
Disallow /design
Disallow /content
Allow /content/dam/cgc/shared-content/sitemap/tcg/esg-sitemap.xml
Allow /content/dam/cgc/shared-content/documents/policies/DEI_Supplier_Code_of_conduct_072523.pdf
Allow /content/dam/*/Images/*
Allow /content/dam/*/images/*
Allow /content/*.css
Allow /content/*.js
Allow /content/*.json
Disallow /global-errors/*
Disallow /*/_jcr_content/*
Allow /etc/designs/default/canvas/content/sites/*
Disallow /institutional/investments/fund/*
Allow /institutional/investments/fund/refgx
Disallow *.pdf

ultraseek

Rule Path
Disallow /advisor
Disallow /individual

twitterbot

Rule Path
Disallow

linkedinbot

Rule Path
Disallow

Other Records

Field Value
sitemap https://www.capitalgroup.com/sitemap_index.xml
sitemap https://www.capitalgroup.com/content/dam/cgc/shared-content/sitemap/tcg/esg-sitemap.xml

Comments

  • robots.txt for http://www.capitalgroup.com
  • Allow S&P
  • exclude ia_archiver and VisWeb robots
  • exclude directories
  • index exclusion for all PDF files
  • exclude advisor content from investor index
  • To enable Twitter and Linkedin access