Log In Sign Up

body-visualizer.com
robots.txt

Robots Exclusion Standard data for body-visualizer.com

Resource Scan

Scan Details

Site Domain body-visualizer.com
Base Domain body-visualizer.com
Scan Status Ok
Last Scan2025-09-10T18:14:01+00:00
Next Scan 2025-09-17T18:14:01+00:00

Last Scan

Scanned2025-09-10T18:14:01+00:00
URL https://body-visualizer.com/robots.txt
Domain IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Response IP 104.21.32.1
Found Yes
Hash 3a098c69506c75c8841f5cc1c6439ba8b603049d8f81716aa7e736e161ede0c5
SimHash 4a14cc526632

Groups

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

yandex

Rule Path
Allow /

Other Records

Field Value
crawl-delay 3

baiduspider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 3

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

*

Rule Path
Allow /
Disallow /api/*
Disallow /admin/*
Disallow /wp-*
Disallow /*.php$
Disallow /*.js$
Disallow /*.css$
Disallow /*.json$

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://body-visualizer.com/sitemap.xml
Back to top

Comments

  • 主要搜索引擎爬虫配置
  • 禁止低质量或不必要的爬虫
  • 允许所有其他爬虫，但限制爬取速度
  • 禁止访问管理和API路由
  • 限制爬取频率
  • 为所有爬虫提供Sitemap
Back to top