desertnaturals.me
robots.txt

Robots Exclusion Standard data for desertnaturals.me

Resource Scan

Scan Details

Site Domain desertnaturals.me
Base Domain desertnaturals.me
Scan Status Ok
Last Scan2024-11-17T03:20:24+00:00
Next Scan 2024-11-24T03:20:24+00:00

Last Scan

Scanned2024-11-17T03:20:24+00:00
URL https://desertnaturals.me/robots.txt
Domain IPs 104.19.154.92
Response IP 104.19.154.92
Found Yes
Hash ac2b7b3c603a5532b200b4ec4bf655c06bbf807b36d8138e69fb1ef4faf5ca5f
SimHash c11ecb22d819

Groups

googlebot

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

ccbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

*

Rule Path
Disallow /cart/
Disallow /checkout/
Disallow /my-account/
Disallow /*add-to-cart%3D*
Allow /wp-admin/admin-ajax.php
Allow /wp-content/uploads/
Allow /wp-content/plugins/*.js
Allow /wp-content/themes/*.js
Allow /wp-content/themes/*.css
Allow /wp-content/cache/*.js
Allow /wp-content/cache/*.css
Allow /wp-includes/*.js
Allow /wp-includes/*.png
Allow /wp-content/cache/*.js
Allow /wp-content/cache/*.css

Comments

  • Block Common Crawl