Log In Sign Up

bstshrt.com
robots.txt

Robots Exclusion Standard data for bstshrt.com

Resource Scan

Scan Details

Site Domain bstshrt.com
Base Domain bstshrt.com
Scan Status Ok
Last Scan2025-11-20T17:24:35+00:00
Next Scan 2025-11-27T17:24:35+00:00

Last Scan

Scanned2025-11-20T17:24:35+00:00
URL https://bstshrt.com/robots.txt
Domain IPs 104.21.63.56, 172.67.143.218, 2606:4700:3035::ac43:8fda, 2606:4700:3036::6815:3f38
Response IP 104.21.63.56
Found Yes
Hash a46d4737e679e891afef94e81f1aea8c9b398c71a0db4fca968860c2089a59b7
SimHash 50198a92a633

Groups

*

Rule Path
Allow /
Disallow /api/
Disallow /api/*
Disallow /dashboard/
Disallow /dashboard/*
Disallow /api/dashboard/
Disallow /api/dashboard/*
Disallow /login
Disallow /register
Disallow /forgot-password
Disallow /reset-password
Disallow /admin/
Disallow /admin/*
Allow /user/
Allow /user/*
Disallow /debug
Disallow /testing
Disallow /preview/
Disallow /preview/*
Allow /sitemap.xml

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /
Disallow /api/
Disallow /api/*
Disallow /dashboard/
Disallow /dashboard/*
Disallow /api/dashboard/
Disallow /api/dashboard/*
Disallow /login
Disallow /register
Disallow /forgot-password
Disallow /admin/
Disallow /admin/*
Allow /user/
Allow /user/*

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /
Disallow /api/
Disallow /api/*
Disallow /dashboard/
Disallow /dashboard/*
Disallow /api/dashboard/
Disallow /api/dashboard/*
Disallow /login
Disallow /register
Disallow /forgot-password
Disallow /admin/
Disallow /admin/*
Allow /user/
Allow /user/*

Other Records

Field Value
crawl-delay 2

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://bstshrt.com/sitemap.xml
Back to top

Comments

  • Disallow ALL API routes
  • Disallow private user areas
  • Disallow authentication pages
  • Disallow admin areas
  • Allow user profile areas for SEO
  • Disallow debug and testing areas
  • Disallow preview areas
  • Allow sitemap
  • Crawl delay (optional)
  • Specific rules for major search engines
  • Block problematic bots
  • Sitemap
Back to top