Log In Sign Up

criticalthinking.com
robots.txt

Robots Exclusion Standard data for criticalthinking.com

Resource Scan

Scan Details

Site Domain criticalthinking.com
Base Domain criticalthinking.com
Scan Status Ok
Last Scan2025-08-16T21:49:54+00:00
Next Scan 2025-09-15T21:49:54+00:00

Last Scan

Scanned2025-08-16T21:49:54+00:00
URL https://criticalthinking.com/robots.txt
Redirect https://www.criticalthinking.com/robots.txt
Redirect Domain www.criticalthinking.com
Redirect Base criticalthinking.com
Domain IPs 192.124.249.56
Redirect IPs 192.124.249.56
Response IP 192.124.249.56
Found Yes
Hash 40c60098632151eab56b8ad7252ed71a6dd18e3f5f0c8f8435d85f1a50729693
SimHash 85017176c2b5

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

googlebot
googlebot-image

Rule Path Comment
Disallow /admin/ -
Disallow /*? -
Disallow /*?q=* -
Disallow /*?*q=* -
Allow /*?page= -
Allow /*?p= -
Allow /*?utm_source= -
Allow /*?utm_medium= -
Allow /*?utm_campaign= -
Allow /*?utm_content= -
Allow /graphql? -
Disallow /repo/ -
Disallow /catalog/product_compare/ -
Disallow /catalog/category/view/ -
Disallow /catalog/product/view/ -
Disallow /catalog/seo_sitemap/ -
Disallow /catalogsearch/ -
Disallow /mfproductsearch/ -
Disallow /control/ -
Disallow /customer/ -
Disallow /ajaxcart/ -
Disallow /ajax/ -
Disallow /quickview/ -
Disallow /productalert/ -
Disallow /mfcmsdr/ -
Disallow /sales/guest/form/ -
Disallow /downloadable/ -
Disallow /pslogin -
Disallow /subscription -
Disallow /newsletter -
Disallow /push_notification -
Disallow /rest/ -
Disallow /app/ -
Disallow /bin/ -
Disallow /dev/ -
Disallow /lib/ -
Disallow /phpserver/ -
Disallow /pkginfo/ -
Disallow /report/ -
Disallow /setup/ -
Disallow /update/ -
Disallow /var/ -
Disallow /vendor/ Paths (clean URLs)
Disallow /catalog/product_compare/ -
Disallow /catalog/category/view/ -
Disallow /catalog/product/view/ -
Disallow /catalogsearch/ -
Disallow /onestepcheckout/ -
Disallow /checkout/ -
Disallow /control/ -
Disallow /contacts/ -
Disallow /customer/ -
Disallow /customer/account/ -
Disallow /customer/account/login/ -
Disallow /customer/account/create/ -
Disallow /customize/ -
Disallow /newsletter/ -
Disallow /review/ -
Disallow /sendfriend/ -
Disallow /wishlist/ -
Disallow /search/term/popular/ -
Disallow /sales/guest/form/ Files
Disallow /cron.php -
Disallow /cron.sh -
Disallow /error_log -
Disallow /install.php -
Disallow /LICENSE.html -
Disallow /LICENSE.txt -
Disallow /LICENSE_AFL.txt -
Disallow /STATUS.txt -
Disallow /get.php -
Disallow /app/ -
Disallow /lib/ -
Disallow /*.php$ -
Disallow /pkginfo/ -
Disallow /report/ -
Disallow /var/ -
Disallow /composer.json -
Disallow /composer.lock -
Disallow /CONTRIBUTING.md -
Disallow /CONTRIBUTOR_LICENSE_AGREEMENT.html -
Disallow /COPYING.txt -
Disallow /Gruntfile.js -
Disallow /nginx.conf.sample -
Disallow /package.json -
Disallow /php.ini.sample -
Disallow /RELEASE_NOTES.txt -
Disallow /*?*product_list_mode= -
Disallow /*?*product_list_order= -
Disallow /*?*product_list_limit= -
Disallow /*?*product_list_dir= -
Disallow /*?SID= -
Disallow /*? -
Disallow /*.php$ CVS, SVN directory and dump files
Disallow /*.CVS -
Disallow /*.Zip$ -
Disallow /*.Svn$ -
Disallow /*.Idea$ -
Disallow /*.Sql$ -
Disallow /*.Tgz$ -
Disallow /phpinfo.php CMS Pages
Disallow /privacy-policy-cookie-restriction-mode -
Disallow /no-route -
Disallow /enable-cookies -
Disallow /home -
Back to top

Comments

  • Do not crawl Magento admin page
  • Urls with parameters
  • Technical paths
  • Do not index pages that are sorted or filtered.
  • Do not index session ID
  • Do not index the general technical directories and files on a server
Back to top

Warnings

  • 1 invalid line.
Back to top