unionhardware.co.zw
robots.txt

Robots Exclusion Standard data for unionhardware.co.zw

Resource Scan

Scan Details

Site Domain unionhardware.co.zw
Base Domain unionhardware.co.zw
Scan Status Ok
Last Scan2025-08-30T20:36:26+00:00
Next Scan 2025-09-29T20:36:26+00:00

Last Scan

Scanned2025-08-30T20:36:26+00:00
URL https://unionhardware.co.zw/robots.txt
Domain IPs 192.124.249.32
Response IP 192.124.249.32
Found Yes
Hash fbd3beb3bbf17aa86db4f682abe806443ecde0415c33558b26e5bfc70e7ef18a
SimHash 53addcd3f698

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/*/
Disallow /msasa/wp-admin/
Disallow /msasa/wp-includes/
Disallow /msasa/wp-content/plugins/
Disallow /pomona/wp-admin/
Disallow /pomona/wp-includes/
Disallow /pomona/wp-content/plugins/
Disallow /lab/*
Allow /wp-content/themes/*/css/
Allow /wp-content/themes/*/js/
Allow /wp-content/themes/*/images/
Disallow /cgi-bin/
Disallow /tmp/
Disallow /private/
Disallow /*.sql$
Disallow /*.log$

googlebot

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

chatbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 5

bingbot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

openai-crawler

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

gemini-web

Rule Path
Disallow /

googleother

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://unionhardware.co.zw/sitemap.xml
sitemap https://unionhardware.co.zw/msasa/sitemap_index.xml
sitemap https://unionhardware.co.zw/pomona/sitemap_index.xml

Comments

  • robots.txt for unionhardware.co.zw
  • Updated: July 2025
  • Allow all legitimate search engines with reasonable crawl delay
  • Block WordPress admin areas for security
  • Block common problematic paths
  • Allow major search engines explicitly
  • Block known problematic/aggressive bots
  • Allow AI bots for better future visibility