Log In Sign Up

digitaltekblog.com
robots.txt

Robots Exclusion Standard data for digitaltekblog.com

Resource Scan

Scan Details

Site Domain digitaltekblog.com
Base Domain digitaltekblog.com
Scan Status Ok
Last Scan2025-12-08T09:24:24+00:00
Next Scan 2025-12-15T09:24:24+00:00

Last Scan

Scanned2025-12-08T09:24:24+00:00
URL https://digitaltekblog.com/robots.txt
Domain IPs 104.21.82.102, 172.67.200.40, 2606:4700:3030::ac43:c828, 2606:4700:3032::6815:5266
Response IP 172.67.200.40
Found Yes
Hash 36bf2d44f748b9e15adf9068655e982b7db07d15a81d244fc6c2e7a62f11c07f
SimHash 2b825a20e5a4

Groups

*

Rule Path
Allow /wp-content/uploads/
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/
Disallow /cgi-bin/
Disallow /private/
Disallow /temp/
Disallow /unused-directory/
Disallow /?s=
Disallow /search/
Disallow /*/feed/
Disallow /*/trackback/
Disallow /*/amp/
Disallow /page/
Allow /category/
Allow /tag/
Back to top

Other Records

Field Value
sitemap https://digitaltekblog.com/sitemap_index.xml
Back to top

Comments

  • ===========================================
  • robots.txt for https://digitaltekblog.com
  • Optimized for WordPress + Rank Math + GeneratePress
  • Last Updated: November 2025
  • ===========================================
  • --- Allow essential assets ---
  • --- Block sensitive or internal system directories ---
  • --- Block low-value or duplicate pages ---
  • --- Allow SEO-beneficial sections ---
  • --- Sitemap for search engines ---
  • ===========================================
  • Notes:
  • 1. Fake or spam URLs are now hard-blocked via .htaccess (403/410).
  • 2. This robots.txt keeps crawl budget focused on valid posts and pages.
  • 3. Test in Google Search Console → Robots.txt Tester.
  • ===========================================
Back to top