Log In Sign Up

colgate.com
robots.txt

Robots Exclusion Standard data for colgate.com

Resource Scan

Scan Details

Site Domain colgate.com
Base Domain colgate.com
Scan Status Ok
Last Scan2025-03-12T03:33:42+00:00
Next Scan 2025-04-11T03:33:42+00:00

Last Scan

Scanned2025-03-12T03:33:42+00:00
URL https://www.colgate.com/robots.txt
Domain IPs 23.215.7.25, 23.215.7.30, 2600:1413:b000:1b::17d7:719, 2600:1413:b000:1b::17d7:71e
Response IP 23.215.7.30
Found Yes
Hash da201f0d1be3fb69bc3f6b5ae50c0df23c6465503aff38fbde9b355ba41d4497
SimHash 6921fb31c710

Groups

*

Rule Path
Allow */search
Allow */content/
Disallow */cp-sites/
Allow */cp-sites/oral-care*
Disallow */smartlabel
Disallow */smiles/
Allow */smiles/special-offers
Disallow *?thumbparam*
Disallow *?filter=*
Disallow *?occ_tag=*
Disallow *?wcmmode=*
Disallow *?bvstate=*
Disallow */b2bcontent/
Disallow */cp-sites-aem/
Disallow /*?*
Disallow */search-results
Disallow /dam/cp-sites/aem*/
Disallow /etc.clientlibs*/
Disallow */product-match-tool*/
Disallow */oral-health-quiz*/
Disallow */results-page/

swiftbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://www.colgate.com/images_sitemap.xml
sitemap https://www.colgate.com/en-us/sitemap.xml
sitemap https://www.colgate.com/es-mx/sitemap.xml
sitemap https://www.colgate.com/es-co/sitemap.xml
sitemap https://www.colgate.com/es-ar/sitemap.xml
sitemap https://www.colgate.com/es-py/sitemap.xml
sitemap https://www.colgate.com/es-bo/sitemap.xml
sitemap https://www.colgate.com/es-ec/sitemap.xml
sitemap https://www.colgate.com/es-ve/sitemap.xml
sitemap https://www.colgate.com/es-cl/sitemap.xml
sitemap https://www.colgate.com/es-pe/sitemap.xml
sitemap https://www.colgate.com/fr-ca/sitemap.xml
sitemap https://www.colgate.com/en-ca/sitemap.xml
sitemap https://www.colgate.com/es-do/sitemap.xml
sitemap https://www.colgate.com/es-uy/sitemap.xml
sitemap https://www.colgate.com/en-gb/sitemap.xml
sitemap https://www.colgate.com/tr-tr/sitemap.xml
sitemap https://www.colgate.com/en-za/sitemap.xml
sitemap https://www.colgate.com/es-us/sitemap.xml
sitemap https://www.colgate.com/th-th/sitemap.xml
sitemap https://www.colgate.com/en-in/sitemap.xml
sitemap https://www.colgate.com/en-sa/sitemap.xml
sitemap https://www.colgate.com/en-ph/sitemap.xml
sitemap https://www.colgate.com/pt-pt/sitemap.xml
sitemap https://www.colgate.com/en-my/sitemap.xml
sitemap https://www.colgate.com/en-sg/sitemap.xml
sitemap https://www.colgate.com/ms-my/sitemap.xml
sitemap https://www.colgate.com/ar-sa/sitemap.xml
sitemap https://www.colgate.com/kz-kz/sitemap.xml
sitemap https://www.colgate.com/es-cr/sitemap.xml
sitemap https://www.colgate.com/es-pa/sitemap.xml
sitemap https://www.colgate.com/es-hn/sitemap.xml
sitemap https://www.colgate.com/es-sv/sitemap.xml
sitemap https://www.colgate.com/es-ni/sitemap.xml
sitemap https://www.colgate.com/es-gt/sitemap.xml
Back to top

Comments

  • Standard Excludes
  • ZG Sitemap Excludes
  • Non-ZG Excludes
  • Sitemap Locations
  • Search Engine Crawl Delays
Back to top

Warnings

  • `noindex` is not a known field.
Back to top