Log In Sign Up

startupbros.com
robots.txt

Robots Exclusion Standard data for startupbros.com

Resource Scan

Scan Details

Site Domain startupbros.com
Base Domain startupbros.com
Scan Status Ok
Last Scan2026-01-31T23:07:38+00:00
Next Scan 2026-03-02T23:07:38+00:00

Last Scan

Scanned2026-01-31T23:07:38+00:00
URL https://startupbros.com/robots.txt
Domain IPs 104.18.12.39, 104.18.13.39, 2606:4700::6812:c27, 2606:4700::6812:d27
Response IP 104.18.12.39
Found Yes
Hash 81180c7217485541d13e8fb8712729b013e4fb06e7c30d9383f859e16d87f022
SimHash aa7c530222b0

Groups

*

Rule Path
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-cron.php
Disallow /tag/
Disallow /search/
Disallow /?s=
Disallow /author/
Disallow /comments/
Disallow /feed/
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /wp-json/wp/v2/users
Disallow /get/
Disallow /tools/
Disallow /go/
Disallow /readme.html
Disallow /license.txt
Disallow /wp-config.php
Disallow /*?replytocom=
Disallow /*?doing_wp_cron=
Back to top

Other Records

Field Value
sitemap https://startupbros.com/sitemap_index.xml
Back to top

Comments

  • ============================================================================
  • StartupBros.com Robots.txt
  • Updated: 2026-01-05
  • Strategy: Maximum AI visibility (Option A) - allow all crawlers
  • ============================================================================
  • Allow search engines to properly render pages (Google requirement)
  • Block WordPress admin and system areas
  • Block non-content pages (crawl budget optimization)
  • Block affiliate redirect paths (no SEO value)
  • Block WordPress system files
  • Block query parameters that create duplicate content
  • Sitemap location
Back to top