colgate.com.cn
robots.txt

Robots Exclusion Standard data for colgate.com.cn

Resource Scan

Scan Details

Site Domain colgate.com.cn
Base Domain colgate.com.cn
Scan Status Ok
Last Scan2024-05-19T09:57:11+00:00
Next Scan 2024-06-18T09:57:11+00:00

Last Scan

Scanned2024-05-19T09:57:11+00:00
URL https://colgate.com.cn/robots.txt
Redirect https://www.colgate.com.cn/robots.txt
Redirect Domain www.colgate.com.cn
Redirect Base colgate.com.cn
Domain IPs 34.230.254.189
Redirect IPs 96.17.180.187, 96.17.180.19
Response IP 23.59.168.178
Found Yes
Hash c8d7979e3d0ddca594542c28c3a4cec369a2263194644e45211a7c39440bc3d4
SimHash 2521a230e380

Groups

*

Rule Path
Disallow */search
Disallow */cp-sites/
Allow */cp-sites/oral-care*
Allow */cp-sites/personal-care*
Allow */cp-sites/home-care*
Disallow */smartlabel
Disallow */smiles/
Allow */smiles/special-offers
Disallow */content/
Disallow */search-results

swiftbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

Other Records

Field Value
sitemap https://www.colgate.com.cn/sitemap.xml

Comments

  • internal search
  • Images
  • Disallow Directives