Log In Sign Up

chnbuyer.com
robots.txt

Robots Exclusion Standard data for chnbuyer.com

Resource Scan

Scan Details

Site Domain chnbuyer.com
Base Domain chnbuyer.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2026-02-07T04:16:14+00:00
Next Scan 2026-02-21T04:16:14+00:00

Last Successful Scan

Scanned2026-01-16T03:47:28+00:00
URL http://chnbuyer.com/robots.txt
Domain IPs 183.136.138.173
Response IP 183.136.138.173
Found Yes
Hash 44ea7042f7912e25751cb8c4b855d28c107677ea59f2b457b8036d5de6910ed8
SimHash 73164e6aeaa6

Groups

*

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

openai

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

google-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

anthropic

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

meta-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

ai21

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

bytespider

Product Comment
bytespider TikTok
Rule Path
Disallow /
Disallow /admin/
Disallow /private/
Disallow /api/
Disallow /ajax/
Disallow /user-data/
Back to top

Other Records

Field Value
sitemap https://chnbuyer.com/sitemap.xml
sitemap https://chnbuyer.com/news-sitemap.xml
Back to top

Comments

  • 允许所有搜索引擎爬虫访问公开内容
  • ======== 屏蔽AI训练爬虫 ========
  • OpenAI
  • Google AI
  • Anthropic (Claude)
  • Common Crawl
  • Facebook/Meta AI
  • 其他AI/数据收集爬虫
  • ======== 目录限制 ========
  • 可选：限制特定目录
  • 站点地图
  • 额外指令
  • 建议爬虫不要缓存页面
  • 限制AI训练使用
Back to top

Warnings

  • `cache-control` is not a known field.
  • `host` is not a known field.
  • `x-robots-tag` is not a known field.
Back to top