Log In Sign Up

techcentral.co.za
robots.txt

Robots Exclusion Standard data for techcentral.co.za

Resource Scan

Scan Details

Site Domain techcentral.co.za
Base Domain techcentral.co.za
Scan Status Ok
Last Scan2025-11-13T11:50:30+00:00
Next Scan 2025-12-13T11:50:30+00:00

Last Scan

Scanned2025-11-13T11:50:30+00:00
URL https://techcentral.co.za/robots.txt
Domain IPs 104.26.10.220, 104.26.11.220, 172.67.73.103, 2606:4700:20::681a:adc, 2606:4700:20::681a:bdc, 2606:4700:20::ac43:4967
Response IP 104.26.10.220
Found Yes
Hash a8feb4c8bbb4cb88516444497f547fbad77109a188f8c30e0f6e325becbfdb82
SimHash 470cd0c266fb

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-config.php
Disallow /wp-load.php
Disallow /wp-cron.php
Disallow /wp-links-opml.php
Disallow /wp-mail.php
Disallow /wp-trackback.php
Disallow /wp-activate.php
Disallow /wp-signup.php
Disallow /wp-comments-post.php
Disallow /xmlrpc.php
Disallow /readme.html
Disallow /license.txt
Disallow /*?s=
Disallow /*%26s%3D
Disallow /search/
Disallow /*?preview=true
Disallow /*?replytocom=
Disallow /*/feed/
Disallow /tag/*/feed/
Disallow /category/*/feed/
Disallow /author/*/feed/
Disallow /*/trackback/
Disallow /phpmyadmin/
Disallow /*.sql$
Disallow /*.log$
Disallow /*.bak$
Disallow /*?page=
Disallow /*?paged=

googlebot

Rule Path
Allow /wp-admin/admin-ajax.php

bingbot

Rule Path
Allow /wp-admin/admin-ajax.php

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 86400

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 86400

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 86400
Back to top

Other Records

Field Value
sitemap https://techcentral.co.za/news-sitemap.xml
sitemap https://techcentral.co.za/sitemap_index.xml
sitemap https://techcentral.co.za/post-sitemap1.xml
Back to top

Comments

  • Robots.txt for TechCentral.co.za
  • Optimized for WordPress with WP Fastest Cache, Wordfence, and Cloudflare
  • Block WordPress admin and sensitive areas
  • Block plugin directories (but not their generated content)
  • Block WordPress system files
  • Block search and filter URLs to avoid duplicate content
  • Block feeds (allow main RSS but block category/tag feeds)
  • Block trackback URLs
  • Block phpMyAdmin
  • Block common backup and log files
  • Block pagination parameters to avoid duplicate content
  • Allow admin-ajax for proper functionality
  • Slow down aggressive bots
  • Sitemap locations
Back to top