Log In Sign Up

silfab.com.ar
robots.txt

Robots Exclusion Standard data for silfab.com.ar

Resource Scan

Scan Details

Site Domain silfab.com.ar
Base Domain silfab.com.ar
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan3/6/2025, 4:07:03 AM
Next Scan 6/4/2025, 4:07:03 AM

Last Successful Scan

Scanned10/20/2023, 3:15:39 PM
URL https://silfab.com.ar/robots.txt
Redirect https://www.silfab.com.ar/robots.txt
Redirect Domain www.silfab.com.ar
Redirect Base silfab.com.ar
Domain IPs 104.21.93.12, 172.67.202.68, 2606:4700:3031::ac43:ca44, 2606:4700:3033::6815:5d0c
Redirect IPs 104.21.93.12, 172.67.202.68, 2606:4700:3031::ac43:ca44, 2606:4700:3033::6815:5d0c
Response IP 172.67.202.68
Found Yes
Hash 1776568dfef952edb2182aa14e1e8cc487f3d83504c60db04ef2dcfa1f65bdbe
SimHash 2fa02b39cef1

Groups

*

Rule Path
Disallow *.csv
Disallow *.svn
Disallow *.idea
Disallow *.sql
Disallow *.tgz
Disallow /admin/
Disallow /app/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /lib/
Disallow /pkginfo/
Disallow /shell/
Disallow /var/
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /*?___from_store=*
Disallow /*?cat=*
Disallow /*?q=*
Disallow /*?SID=
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /catalog/product/gallery/
Disallow /javascript/
Disallow /poll/
Disallow /review/
Disallow /sendfriend/
Disallow /tag/
Disallow /wishlist/
Disallow /ajax/
Disallow /ajaxwishlist/
Disallow /priceslider/

Other Records

Field Value
crawl-delay 60
Back to top

Comments

  • Enable robots.txt rules for all crawlers
  • Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
  • Set a custom crawl rate if you're experiencing traffic problems with your server.
  • Magento sitemap: uncomment and replace the URL to your Magento sitemap file
  • Sitemap: http://www.example.com/sitemap/sitemap.xml
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • GENERAL MAGENTO SETTINGS
  • Do not crawl Magento admin page
  • Do not crawl common Magento technical folders
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
  • Disallow: /index.php/
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • Do not crawl seach pages and not-SEO catalog links
  • Do not crawl ajax requests
Back to top