aicodewith.com
robots.txt

Robots Exclusion Standard data for aicodewith.com

Resource Scan

Scan Details

Site Domain aicodewith.com
Base Domain aicodewith.com
Scan Status Ok
Last Scan2025-11-16T03:20:12+00:00
Next Scan 2025-12-16T03:20:12+00:00

Last Scan

Scanned2025-11-16T03:20:12+00:00
URL https://aicodewith.com/robots.txt
Domain IPs 104.21.95.230, 172.67.149.11, 2606:4700:3031::6815:5fe6, 2606:4700:3031::ac43:950b
Response IP 172.67.149.11
Found Yes
Hash 4ff463dc718fea7ead1407c0e63eeb044e0ff6f9eb07524f786d9b085bc4bf9f
SimHash 4c9adcf02713

Groups

*

Rule Path
Allow /
Disallow /api/
Disallow /auth/
Disallow /dashboard/settings
Disallow /dashboard/credits
Disallow /dashboard/subscription
Allow /login
Allow /forgot-password
Allow /dashboard/tutorial
Allow /dashboard/installation

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

Other Records

Field Value
sitemap https://aicodewith.com/sitemap.xml

Comments

  • 禁止访问API路径
  • 禁止访问私有页面
  • 允许访问公开页面
  • 网站地图位置
  • 爬取延迟(毫秒)
  • 针对特定搜索引擎的规则