d.image.i4.cn
robots.txt

Robots Exclusion Standard data for d.image.i4.cn

Resource Scan

Scan Details

Site Domain d.image.i4.cn
Base Domain i4.cn
Scan Status Ok
Last Scan2024-08-31T00:53:25+00:00
Next Scan 2024-09-30T00:53:25+00:00

Last Scan

Scanned2024-08-31T00:53:25+00:00
URL http://d.image.i4.cn/robots.txt
Domain IPs 163.171.208.133, 203.117.159.85
Response IP 163.171.208.133
Found Yes
Hash 0a5c12ffbf766569cacd01537f6de4600475be8dd94f3066030df218a004e546
SimHash c8589a6283f2

Groups

*

Rule Path
Disallow /

Comments

  • -----------------------------------------------------------------------------
  • 禁止爬虫爬取无效URL,提升网站核心静态资源抓取及索引效率。
  • 无效URL包含:全动态URL,重复页面URL,存在问题的旧静态URL
  • 等各种无需被SE收录的URL。
  • -----------------------------------------------------------------------------