Log In Sign Up

backd.vc
robots.txt

Robots Exclusion Standard data for backd.vc

Resource Scan

Scan Details

Site Domain backd.vc
Base Domain backd.vc
Scan Status Ok
Last Scan2025-09-22T09:49:00+00:00
Next Scan 2025-10-22T09:49:00+00:00

Last Scan

Scanned2025-09-22T09:49:00+00:00
URL https://backd.vc/robots.txt
Domain IPs 185.158.133.1
Response IP 185.158.133.1
Found Yes
Hash cb5909a98728f3f1f550210e8583ab8661c510714b6193b23f6d6a748023c44b
SimHash 2d959cd5e5e7

Groups

*

Rule Path
Allow /
Allow /index.html
Allow /about
Allow /startups
Allow /real-estate
Allow /blog
Allow /faq
Allow /investment-guides
Allow /ke/
Allow /kenya-investments
Allow /nairobi-real-estate
Allow /kenya-startups
Disallow /admin/
Disallow /admin/*
Disallow /signin
Disallow /signup
Disallow /kyc
Allow /terms
Allow /privacy-policy
Allow /cookie-policy
Allow /risk
Allow /disclosures

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0.5

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

twitterbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

yandexbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://backd.co/sitemap.xml
sitemap https://backd.co/ke/sitemap.xml
Back to top

Comments

  • Back'd robots.txt - Enhanced for SEO
  • Important pages - Using specific directives for important content
  • Kenya-specific pages
  • Block access to admin routes
  • Block access to auth routes (prevent crawling of login/signup pages)
  • Allow access to legal pages
  • Crawl delay for responsible crawling
  • Sitemaps
  • Social media bots - with specific directives
  • Regional search engines
Back to top