Log In Sign Up

rokhardware.com
robots.txt

Robots Exclusion Standard data for rokhardware.com

Resource Scan

Scan Details

Site Domain rokhardware.com
Base Domain rokhardware.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-01-28T21:11:47+00:00
Next Scan 2026-04-28T21:11:47+00:00

Last Successful Scan

Scanned2024-09-13T09:02:51+00:00
URL https://rokhardware.com/robots.txt
Redirect https://www.rokhardware.com/robots.txt
Redirect Domain www.rokhardware.com
Redirect Base rokhardware.com
Domain IPs 104.26.6.125, 104.26.7.125, 172.67.70.48, 2606:4700:20::681a:67d, 2606:4700:20::681a:77d, 2606:4700:20::ac43:4630
Redirect IPs 104.26.6.125, 104.26.7.125, 172.67.70.48, 2606:4700:20::681a:67d, 2606:4700:20::681a:77d, 2606:4700:20::ac43:4630
Response IP 172.67.70.48
Found Yes
Hash cbaa5a3f4104a780df77178033c507d1dede086f90fd4f9cc7c31367ddb64f2c
SimHash 6f602b1566f3

Groups

*

Rule Path
Disallow CVS
Disallow .svn
Disallow .idea
Disallow .sql
Disallow .tgz
Disallow /admin/
Disallow /app/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /lib/
Disallow /pkginfo/
Disallow /shell/
Disallow /var/
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /*?___from_store=*
Disallow /*?cat=*
Disallow /*?q=*
Disallow /*?p=1*
Disallow /*?dir=*
Disallow /*%26limit%3D*
Disallow /index.php/
Disallow /*?SID=
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /catalog/product/gallery/
Disallow /poll/
Disallow /review/
Disallow /sendfriend/
Disallow /tag/
Disallow /wishlist/
Disallow /ajax/
Disallow /ajaxwishlist/
Disallow /priceslider/

Other Records

Field Value
crawl-delay 10

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://www.rokhardware.com/media/sitemap/sitemap.xml
Back to top

Comments

  • Enable robots.txt rules for all crawlers
  • Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
  • Set a custom crawl rate if you're experiencing traffic problems with your server.
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • GENERAL MAGENTO SETTINGS
  • Do not crawl Magento admin page
  • Do not crawl common Magento technical folders
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • Do not crawl seach pages and not-SEO catalog links
  • Do not crawl ajax requests
Back to top