Log In Sign Up

ale8one.com
robots.txt

Robots Exclusion Standard data for ale8one.com

Resource Scan

Scan Details

Site Domain ale8one.com
Base Domain ale8one.com
Scan Status Ok
Last Scan5/4/2025, 6:23:16 AM
Next Scan 6/3/2025, 6:23:16 AM

Last Scan

Scanned5/4/2025, 6:23:16 AM
URL https://ale8one.com/robots.txt
Domain IPs 104.21.36.253, 172.67.201.161, 2606:4700:3033::6815:24fd, 2606:4700:3036::ac43:c9a1
Response IP 104.21.36.253
Found Yes
Hash daf1fa8fb6860399d40787a79ef7e40f5c77775868f94f06c7047b33b3446a0f
SimHash af65496966e7

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /feed/
Disallow /store/CVS
Disallow /store/*.svn$
Disallow /store/*.idea$
Disallow /store/*.sql$
Disallow /store/*.tgz$
Disallow /store/admin/
Disallow /store/xlogin/
Disallow /store/app/
Disallow /store/downloader/
Disallow /store/errors/
Disallow /store/includes/
Disallow /store/lib/
Disallow /store/pkginfo/
Disallow /store/shell/
Disallow /store/var/
Disallow /store/404/
Disallow /store/cgi-bin/
Disallow /store/js/
Disallow /store/lib/
Disallow /store/magento/
Disallow /store/report/
Disallow /store/scripts/
Disallow /store/stats/
Disallow /store/api.php
Disallow /store/cron.php
Disallow /store/cron.sh
Disallow /store/error_log
Disallow /store/get.php
Disallow /store/install.php
Disallow /store/LICENSE.html
Disallow /store/LICENSE.txt
Disallow /store/LICENSE_AFL.txt
Disallow /store/README.txt
Disallow /store/RELEASE_NOTES.txt
Disallow /store/STATUS.txt
Disallow /store/*?dir*
Disallow /store/*?dir=desc
Disallow /store/*?dir=asc
Disallow /store/*?limit=all
Disallow /store/*?mode*
Disallow /store/index.php/
Disallow /store/*?SID=
Disallow /store/checkout/
Disallow /store/onestepcheckout/
Disallow /store/customer/
Disallow /store/customer/account/
Disallow /store/customer/account/login/
Disallow /store/catalogsearch/
Disallow /store/catalog/product_compare/
Disallow /store/catalog/category/view/
Disallow /store/catalog/product/view/
Disallow /store/catalog/product/gallery/
Disallow /store/control/
Disallow /store/customize/
Disallow /store/newsletter/
Disallow /store/poll/
Disallow /store/sendfriend/
Disallow /store/shopby/
Disallow /store/tag/
Disallow /store/wishlist/
Disallow /store/cgi-bin/
Disallow /store/cleanup.php
Disallow /store/apc.php
Disallow /store/memcache.php
Disallow /store/phpinfo.php
Back to top

Comments

  • Enable robots.txt rules for all crawlers
  • WORDPRESS
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • GENERAL MAGENTO SETTINGS
  • Do not crawl Magento admin page
  • Do not crawl common Magento technical folders
  • Disallow: /store/media/
  • Disallow: /store/skin/
  • Do not crawl common Magento files
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • Do not crawl seach pages and not-SEO optimized catalog links
  • Do not default Magento tools
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
  • IMAGE CRAWLERS SETTINGS
  • Extra: Uncomment if you do not wish Google and Bing to index your images
  • User-agent: Googlebot-Image
  • Disallow: /
  • User-agent: msnbot-media
  • Disallow: /
Back to top