Log In Sign Up

justelectronics.co.za
robots.txt

Robots Exclusion Standard data for justelectronics.co.za

Resource Scan

Scan Details

Site Domain justelectronics.co.za
Base Domain justelectronics.co.za
Scan Status Ok
Last Scan2025-09-18T10:30:02+00:00
Next Scan 2025-09-25T10:30:02+00:00

Last Scan

Scanned2025-09-18T10:30:02+00:00
URL https://justelectronics.co.za/robots.txt
Domain IPs 104.21.7.52, 172.67.135.197, 2606:4700:3030::6815:734, 2606:4700:3035::ac43:87c5
Response IP 172.67.135.197
Found Yes
Hash 88dacc1d7c4ca3042503cd1264ddb7b5090d695b8ff964c476f60a4c00b3d5e7
SimHash 6a16a89be0eb

Groups

*

Rule Path
Disallow /wp-content/uploads/wc-logs/
Disallow /wp-content/uploads/woocommerce_transient_files/
Disallow /wp-content/uploads/woocommerce_uploads/
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /search/
Disallow /tag/
Disallow /category/
Disallow /author/
Disallow /feed/
Disallow /rss/
Disallow /*orderby%3D
Disallow /*min_price%3D
Disallow /*max_price%3D
Disallow /*rating%3D
Disallow /*filter_
Disallow /*?add-to-cart=*
Disallow /*?add_to_wishlist=*
Disallow /*?orderby=*
Disallow /*?_wpnonce=*
Disallow /*?query=*
Disallow /*?s=*
Disallow /*?coupon_code=*
Disallow /*?removed_item=*
Disallow /*?wc-ajax=*

gptbot

Rule Path
Allow /

ccbot

Rule Path
Allow /

claudebot

Rule Path
Allow /

claude-user

Rule Path
Allow /

claude-searchbot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

google-extended

Rule Path
Allow /

applebot-extended

Rule Path
Allow /

amazonbot

Rule Path
Allow /

bingbot

Rule Path
Allow /

bingpreview

Rule Path
Allow /

geedoproductsearch

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://justelectronics.co.za/sitemap_index.xml
sitemap https://justelectronics.co.za/news-sitemap.xml
Back to top

Comments

  • -----------------------------
  • Just Electronics - robots.txt
  • -----------------------------
  • XML Sitemaps
  • Opt-in for AI crawlers
  • -------------------------------------------------
  • Block server logs and temp files
  • Block admin area (except AJAX)
  • Block search, tag, category and feed pages to reduce duplicate content
  • Remove noisy URL parameters
  • --- Invite reputable AI crawlers explicitly --------------------
  • Google’s AI product token (not a search crawler)
  • Apple’s AI training token (separate from Applebot search crawler)
  • Amazonbot (for Amazon services and search touchpoints)
  • Microsoft Copilot/Bing AI invitation
  • --- Throttle known price-scraping bot --------
  • (Keeps it polite without fully blocking. Increase delay or Disallow if needed.)
  • END OF FILE
Back to top

Warnings

  • `llms` is not a known field.
Back to top