Log In Sign Up

naturitas.es
robots.txt

Robots Exclusion Standard data for naturitas.es

Resource Scan

Scan Details

Site Domain naturitas.es
Base Domain naturitas.es
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-01-20T01:27:38+00:00
Next Scan 2026-02-19T01:27:38+00:00

Last Successful Scan

Scanned2025-12-13T22:54:50+00:00
URL https://naturitas.es/robots.txt
Redirect https://www.naturitas.es/robots.txt
Redirect Domain www.naturitas.es
Redirect Base naturitas.es
Domain IPs 104.18.26.106, 104.18.27.106, 2606:4700::6812:1a6a, 2606:4700::6812:1b6a
Redirect IPs 104.18.26.106, 104.18.27.106, 2606:4700::6812:1a6a, 2606:4700::6812:1b6a
Response IP 104.18.26.106
Found Yes
Hash b55d941951f0acf99b99c318b57e4d107b27fb86b9a616649ae6833f2e3dd245
SimHash 7134701141db

Groups

*

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /
Disallow /cdn-cgi/
Disallow /*.php$
Disallow /pkginfo/
Disallow /report/
Disallow /sendfriend/
Disallow /sendfriend/product/
Disallow /wishlist/
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /pub/
Disallow /var/
Disallow /rest/V1/promise/product
Disallow /checkout/
Disallow /checkout/cart/
Disallow /checkout/onepage/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /myaccount/orders/
Disallow /wishlist/
Disallow /terminos-condiciones/
Disallow /politica-devoluciones/
Disallow /catalog/product_compare/
Disallow /catalog/product_compare/add/
Disallow /catalog/product_compare/index/
Disallow /catalogsearch/
Disallow /catalogsearch/result/
Disallow /tag/
Disallow /review/
Disallow /composer.json
Disallow /composer.lock
Disallow /CONTRIBUTING.md
Disallow /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow /COPYING.txt
Disallow /Gruntfile.js
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /nginx.conf.sample
Disallow /package.json
Disallow /php.ini.sample
Disallow /RELEASE_NOTES.txt
Disallow /*.git
Disallow /*.CVS
Disallow /*.Zip$
Disallow /*.Svn$
Disallow /*.Idea$
Disallow /*.Sql$
Disallow /*.Tgz$
Disallow /*?discount_rate=
Disallow /*?essences=
Disallow /*?flags=
Disallow /*?flavours=
Disallow /*?price=
Disallow /*?uses=
Disallow /*?color=
Disallow /*?rcnOrigin=
Disallow /*?seals=
Disallow /*?nosto=
Disallow /*?refSrc=
Disallow /*?product_list_order=
Disallow /*?manufacturer=
Disallow /*?brand=
Disallow /*?product_list_dir=
Disallow /*?brand_category=

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

seekportbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

duckduckbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

facebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

sentibot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

sentibot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

geedoproductsearch

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

amazonbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.naturitas.es/sitemap.xml
Back to top

Comments

  • Disable CF folder
  • Disable common folders
  • Disable Promise Endpoints
  • Disable checkout & customer account
  • Disable customer orders, wishlist & some footer pages
  • Disable Search pages & catalog direct routing
  • Disable Tag & Review (Avoid duplicate content)
  • Common files
  • Disable version control folders and others
  • Disable filters parameters
  • Waiting time between calls with most common bots
  • Block bots
Back to top