Log In Sign Up

bloksec.com
robots.txt

Robots Exclusion Standard data for bloksec.com

Resource Scan

Scan Details

Site Domain bloksec.com
Base Domain bloksec.com
Scan Status Ok
Last Scan2025-06-21T02:29:44+00:00
Next Scan 2025-07-21T02:29:44+00:00

Last Scan

Scanned2025-06-21T02:29:44+00:00
URL https://bloksec.com/robots.txt
Domain IPs 34.111.179.208
Response IP 34.111.179.208
Found Yes
Hash f98f1311ef018e599dc930baa4bc0f8c233acbc5ac747d841e6c566b5056523c
SimHash ec277eb3e7c5

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /_next/
Disallow /node_modules/
Allow /assets/
Allow /images/
Allow /css/
Allow /js/

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://bloksec.com/sitemap.xml
Back to top

Comments

  • Allow all search engines to crawl the site
  • Disallow crawling of API endpoints
  • Disallow crawling of admin or internal pages
  • Allow crawling of static assets
  • Sitemap location
  • Crawl delay (optional - be respectful to smaller crawlers)
Back to top