colgate.com.cn
robots.txt

Robots Exclusion Standard data for colgate.com.cn

Resource Scan

Scan Details

Site Domain colgate.com.cn
Base Domain colgate.com.cn
Scan Status Ok
Last Scan2024-06-18T09:57:25+00:00
Next Scan 2024-07-18T09:57:25+00:00

Last Scan

Scanned2024-06-18T09:57:25+00:00
URL https://colgate.com.cn/robots.txt
Redirect https://www.colgate.com.cn/robots.txt
Redirect Domain www.colgate.com.cn
Redirect Base colgate.com.cn
Domain IPs 34.230.254.189
Redirect IPs 184.28.235.73, 184.28.235.75
Response IP 23.32.29.8
Found Yes
Hash c8d7979e3d0ddca594542c28c3a4cec369a2263194644e45211a7c39440bc3d4
SimHash 2521a230e380

Groups

*

Rule Path
Disallow */search
Disallow */cp-sites/
Allow */cp-sites/oral-care*
Allow */cp-sites/personal-care*
Allow */cp-sites/home-care*
Disallow */smartlabel
Disallow */smiles/
Allow */smiles/special-offers
Disallow */content/
Disallow */search-results

swiftbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

Other Records

Field Value
sitemap https://www.colgate.com.cn/sitemap.xml

Comments

  • internal search
  • Images
  • Disallow Directives