Log In Sign Up

toolorbit.com
robots.txt

Robots Exclusion Standard data for toolorbit.com

Resource Scan

Scan Details

Site Domain toolorbit.com
Base Domain toolorbit.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-09T02:51:41+00:00
Next Scan 2026-01-07T02:51:41+00:00

Last Successful Scan

Scanned2023-12-20T02:48:51+00:00
URL https://toolorbit.com/robots.txt
Domain IPs 104.26.6.156, 104.26.7.156, 172.67.72.241, 2606:4700:20::681a:69c, 2606:4700:20::681a:79c, 2606:4700:20::ac43:48f1
Response IP 104.26.6.156
Found Yes
Hash 070a05eb00e309aa898ccf82880775b4347672287b91607dea81da5f3851b6c6
SimHash a5606d5cc6f1

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

googlebot-image

Rule Path
Allow /
Disallow /CVS
Disallow /*.svn$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$
Disallow */admin/
Disallow /app/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /lib/
Disallow /pkginfo/
Disallow /shell/
Disallow /var/
Disallow /cp2020admin/
Disallow /api.php
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /get.php
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /README.txt
Disallow /RELEASE_NOTES.txt
Disallow */*?dir*
Disallow */*?dir=desc
Disallow */*?dir=asc
Disallow */*?limit=all
Disallow */*?mode*
Disallow */*?p=*&
Disallow /*.js$
Disallow /*.css$
Disallow /*.php$
Disallow */index.php/
Disallow */*?SID=
Disallow */checkout/
Disallow */onestepcheckout/
Disallow */customer/
Disallow */customer/account/
Disallow */customer/account/login/
Disallow */catalogsearch/
Disallow */catalog/product_compare/
Disallow */catalog/category/view/
Disallow */catalog/product/view/
Disallow */terms-of-sale/
Disallow */terms-of-use/
Disallow */privacy-policy/
Disallow */sales/order/
Disallow */storecredit/info/
Disallow */invitation/
Disallow /cgi-bin/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php

Other Records

Field Value
crawl-delay 20
Back to top

Comments

  • Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
  • Set a custom crawl rate if you're experiencing traffic problems with your server.
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • GENERAL MAGENTO SETTINGS
  • Do not crawl Magento admin page
  • Do not crawl common Magento technical folders
  • Do not crawl common Magento files
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Disallow: */*?
  • Paths (no clean URLs)
  • Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • Do not crawl search pages and not-SEO optimized catalog links
  • Do not crawl some specific CMS pages & others
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
Back to top