shengyidi.com
robots.txt

Robots Exclusion Standard data for shengyidi.com

Resource Scan

Scan Details

Site Domain shengyidi.com
Base Domain shengyidi.com
Scan Status Ok
Last Scan2024-06-13T22:37:29+00:00
Next Scan 2024-07-13T22:37:29+00:00

Last Scan

Scanned2024-06-13T22:37:29+00:00
URL https://shengyidi.com/robots.txt
Redirect https://www.shengyidi.com/robots.txt
Redirect Domain www.shengyidi.com
Redirect Base shengyidi.com
Domain IPs 39.105.71.234
Redirect IPs 61.160.192.101
Response IP 58.218.215.168
Found Yes
Hash 3ce9cccd0f21bce4b4faadae1b03e99d1af14a35115fc67d371deefba5aaf81b
SimHash 7c46db7dc5b6

Groups

*

Rule Path
Disallow /category/
Disallow /sale/
Disallow /chang/
Disallow /purchase/
Disallow /proxy/
Disallow /exhibition/
Disallow /news/
Disallow /zhuangxiu/
Disallow /hub/
Disallow /recommend/
Disallow /buyer/
Disallow /peixun/
Disallow /liuxue/

Comments

  • -----------------------------------------------------------------------------
  • fileEncoding = UTF-8
  • 禁止爬虫爬取无效URL。
  • 无效URLåŒ
  • 等各种无需被SE收录的URL。
  • -----------------------------------------------------------------------------

Warnings

  • 1 invalid line.
  • `含` is not a known field.