Log In Sign Up

jefferiessocks.com
robots.txt

Robots Exclusion Standard data for jefferiessocks.com

Resource Scan

Scan Details

Site Domain jefferiessocks.com
Base Domain jefferiessocks.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-06-14T08:12:37+00:00
Next Scan 2025-09-12T08:12:37+00:00

Last Successful Scan

Scanned2023-02-02T16:29:36+00:00
URL https://jefferiessocks.com/robots.txt
Redirect https://www.jefferiessocks.com/robots.txt
Redirect Domain www.jefferiessocks.com
Redirect Base jefferiessocks.com
Domain IPs 104.21.37.178, 172.67.211.66, 2606:4700:3032::ac43:d342, 2606:4700:3036::6815:25b2
Redirect IPs 104.21.37.178, 172.67.211.66, 2606:4700:3032::ac43:d342, 2606:4700:3036::6815:25b2
Response IP 104.21.37.178
Found Yes
Hash f1bc5a48142bd0b13c3c340c9503c948d0d2603bca7326ef019b63a38808233d
SimHash a768f55507f3

Groups

*

Rule Path
Disallow /404/
Disallow /app/
Disallow /cgi-bin/
Disallow /downloader/
Disallow /includes/
Disallow /js/
Disallow /lib/
Disallow /magento/
Disallow /pkginfo/
Disallow /report/
Disallow /skin/
Disallow /stats/
Disallow /var/
Disallow /CVS
Disallow /*.git$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$
Disallow */admin/
Disallow /errors/
Disallow /shell/
Disallow /api.php
Disallow /get.php
Disallow /README.txt
Disallow /RELEASE_NOTES.txt
Disallow */*?
Disallow */*?dir*
Disallow */*?dir=desc
Disallow */*?dir=asc
Disallow */*?limit=all
Disallow */*?mode*
Disallow */*?p=*&
Disallow /index.php/
Disallow /control/
Disallow /contacts/
Disallow /customize/
Disallow /newsletter/
Disallow /poll/
Disallow /review/
Disallow /sendfriend/
Disallow /tag/
Disallow /wishlist/
Disallow */checkout/
Disallow */onestepcheckout/
Disallow */customer/
Disallow */customer/account/
Disallow */customer/account/login/
Disallow */catalog/product_compare/
Disallow */sales/order/
Disallow */storecredit/info/
Disallow */invitation/
Disallow /directory/currency/*
Disallow /checkout/*
Disallow /catalogsearch/*
Disallow /paypal/
Disallow /paypal/*
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /STATUS.txt
Disallow /?SID
Disallow /xcart/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php
Back to top

Other Records

Field Value
sitemap https://www.jefferiessocks.com/sitemap.xml
Back to top

Comments

  • Shero robots.txt file for Magento
  • PAGES
  • Disallow: /privacy-policy-cookie-restriction-mode/
  • Disallow: /terms/
  • MAGENTO DIRECTORIES & FILES
  • Directories
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • GENERAL MAGENTO SETTINGS
  • Do not crawl Magento admin page
  • Do not crawl common Magento technical folders
  • Do not crawl common Magento files
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Paths (clean URLs)
  • Do not crawl checkout and user account pages
  • Do not crawl search pages and not-SEO optimized catalog links
  • Disallow: */catalogsearch/
  • Disallow: */catalog/category/view/
  • Disallow: */catalog/product/view/
  • Do not crawl some specific CMS pages & others
  • Disallow: */terms-of-sale/
  • Disallow: */terms-of-use/
  • Disallow: */privacy-policy/
  • Bing still attempts to index these
  • Files
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
Back to top