Log In Sign Up

bitrabo.com
robots.txt

Robots Exclusion Standard data for bitrabo.com

Resource Scan

Scan Details

Site Domain bitrabo.com
Base Domain bitrabo.com
Scan Status Ok
Last Scan2025-09-17T03:16:40+00:00
Next Scan 2025-09-24T03:16:40+00:00

Last Scan

Scanned2025-09-17T03:16:40+00:00
URL https://bitrabo.com/robots.txt
Redirect https://www.bitrabo.com/robots.txt
Redirect Domain www.bitrabo.com
Redirect Base bitrabo.com
Domain IPs 104.21.67.22, 172.67.167.200, 2606:4700:3030::ac43:a7c8, 2606:4700:3033::6815:4316
Redirect IPs 104.21.67.22, 172.67.167.200, 2606:4700:3030::ac43:a7c8, 2606:4700:3033::6815:4316
Response IP 104.21.67.22
Found Yes
Hash d4599f42f3b9a244bee2dadb42f6c4aeb32e52622de77a86c2657b201efb86bb
SimHash 6144c190ef72

Groups

*

Rule Path Comment
Disallow /discover/feed/ -
Disallow /discover/comments/ -
Disallow /discover/*?attachment_id= -
Disallow /discover/wp-json/ -
Disallow /discover/?s= WordPress search
Allow /discover/tag/ -
Allow /discover/author/ -
Allow /discover/page/ -
Allow /discover/creator/ -
Allow /discover/wp-content/uploads/ -
Disallow /discover/wp-admin/ -
Back to top

Other Records

Field Value
sitemap https://www.bitrabo.com/discover/sitemap_index.xml
sitemap https://www.bitrabo.com/sitemap.xml
Back to top

Comments

  • Prevent crawl waste (feeds, search, attachments)
  • Optional: Allow crawl of tag, author, and pagination for now (better to noindex via meta tag later)
  • Allow uploads
  • Disallow sensitive or private areas if any (uncomment if needed)
  • Sitemap for WordPress (media/news)
  • Sitemap for main app/pages
Back to top