Log In Sign Up

sic.co.ke
robots.txt

Robots Exclusion Standard data for sic.co.ke

Resource Scan

Scan Details

Site Domain sic.co.ke
Base Domain sic.co.ke
Scan Status Ok
Last Scan2025-12-02T19:46:54+00:00
Next Scan 2026-01-01T19:46:54+00:00

Last Scan

Scanned2025-12-02T19:46:54+00:00
URL https://sic.co.ke/robots.txt
Domain IPs 41.80.37.36
Response IP 41.80.37.36
Found Yes
Hash aef71efdf55897e5a81dad1396945cd0f057faaccf86596ad0adf8a1492d7776
SimHash 4e5fdd41e5b2

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/

googlebot

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/

Other Records

Field Value
crawl-delay 10

bingbot

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/

Other Records

Field Value
crawl-delay 10

serpstatbot

Rule Path
Disallow /

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/cache/
Disallow /wp-content/themes/
Disallow /wp-json/
Disallow /xmlrpc.php
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /wp-config.php
Disallow /readme.html
Disallow /license.txt
Disallow /wp-trackback.php
Disallow /feed/
Disallow /comments/feed/
Disallow /trackback/
Disallow /*?*
Disallow /*?
Disallow /search/
Disallow /?s=
Disallow /author/

Other Records

Field Value
crawl-delay 5

googlebot

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/cache/
Disallow /wp-json/
Disallow /xmlrpc.php
Disallow /wp-login.php
Disallow /*?*
Allow /wp-content/uploads/

Other Records

Field Value
crawl-delay 3

bingbot

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/cache/
Disallow /wp-json/
Disallow /xmlrpc.php
Disallow /wp-login.php
Disallow /*?*
Allow /wp-content/uploads/

Other Records

Field Value
crawl-delay 5

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/

Other Records

Field Value
crawl-delay 10

facebookexternalhit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://yoursite.com/sitemap.xml
Back to top

Comments

  • Enhanced robots.txt for WordPress
  • Last updated: 2025
  • Block all bots from sensitive WordPress areas
  • Allow major search engines with specific rules
  • Block aggressive SEO/marketing bots
  • Block social media bots that can be resource intensive
  • Block AI training bots (optional - remove if you want AI to crawl)
  • Sitemap location (update with your actual sitemap URL)
Back to top