Log In Sign Up

caravelascolecoes.com.br
robots.txt

Robots Exclusion Standard data for caravelascolecoes.com.br

Resource Scan

Scan Details

Site Domain caravelascolecoes.com.br
Base Domain caravelascolecoes.com.br
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-08-30T03:49:16+00:00
Next Scan 2024-11-28T03:49:16+00:00

Last Successful Scan

Scanned2023-10-12T05:24:50+00:00
URL https://caravelascolecoes.com.br/robots.txt
Domain IPs 104.21.34.105, 172.67.159.26, 2606:4700:3033::6815:2269, 2606:4700:3033::ac43:9f1a
Response IP 172.67.159.26
Found Yes
Hash 9e5af5ce4054bad742f3e3d8ffdfbf0a8cbda3ca7271e8de98c98063d418922e
SimHash 2d706311c6f1

Groups

*

Rule Path
Disallow /CVS
Disallow /*.svn$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$

baiduspider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-imagem

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-news

Rule Path
Disallow /

baiduspider-favo

Rule Path
Disallow /

baiduspider-cpro

Rule Path
Disallow /

baiduspider-anuncios

Rule Path
Disallow /

baidu

Rule Path
Disallow /

majestic12

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yandex

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /
Disallow /admin/
Disallow /app/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /lib/
Disallow /pkginfo/
Disallow /shell/
Disallow /var/
Disallow /api.php
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /get.php
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /README.txt
Disallow /RELEASE_NOTES.txt
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /*?SID=
Disallow /lojav1/
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /cgi-bin/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php

googlebot

Rule Path
Disallow

slurp

Rule Path
Disallow

msnbot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

yahoo-mmcrawler

Rule Path
Disallow

psbot

Rule Path
Disallow
Disallow /review/
Disallow /productpageshipping/
Back to top

Comments

  • robots.txt for Magento Community and Enterprise
  • GENERAL SETTINGS
  • Enable robots.txt rules for all crawlers
  • Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
  • Set a custom crawl rate if you're experiencing traffic problems with your server.
  • Crawl-delay: 30
  • Magento sitemap: uncomment and replace the URL to your Magento sitemap file
  • Sitemap: https://www.caravelascolecoes.com.br/sitemap/sitemap.xml
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • Baiduspider
  • GENERAL MAGENTO SETTINGS
  • Do not crawl Magento admin page
  • Do not crawl common Magento technical folders
  • Do not crawl common Magento files
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
  • Disallow: /index.php/
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • Do not crawl seach pages and not-SEO optimized catalog links
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
  • IMAGE CRAWLERS SETTINGS
  • Extra: Uncomment if you do not wish Google and Bing to index your images
Back to top