Log In Sign Up

buildersmart.in
robots.txt

Robots Exclusion Standard data for buildersmart.in

Resource Scan

Scan Details

Site Domain buildersmart.in
Base Domain buildersmart.in
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-11-04T11:31:22+00:00
Next Scan 2026-02-02T11:31:22+00:00

Last Successful Scan

Scanned2025-04-02T06:29:27+00:00
URL https://buildersmart.in/robots.txt
Domain IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Response IP 104.21.96.1
Found Yes
Hash 717b484ac1035793d5a61b3be59f16e9c0a2ea3fb9274da069538e83ca8cf244
SimHash af602f5346e1

Groups

archive.org_bot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

*

Rule Path
Disallow /CVS
Disallow /*.svn$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$
Disallow /bmpanel/
Disallow /app/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /lib/
Disallow /pkginfo/
Disallow /shell/
Disallow /var/
Disallow /api/
Disallow /CRMImages/
Disallow /custom/
Disallow /STORES/
Disallow /staging/
Disallow /tmp/
Disallow /test/
Disallow /powerbi/
Disallow /Mobile-Detect/
Disallow /media/captcha/
Disallow /media/customer/
Disallow /media/dhl/
Disallow /media/downloadable/
Disallow /media/import/
Disallow /media/pdf/
Disallow /media/sales/
Disallow /media/tmp/
Disallow /api.php
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /get.php
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /README.txt
Disallow /RELEASE_NOTES.txt
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /*category%3D*
Disallow /*aw_shopbybrand_brand%3D*
Disallow /*price%3D*
Disallow /*order%3D*
Disallow /index.php/
Disallow /*?SID=
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalogsearch/
Disallow /*limit%3D*
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /videos/search/*
Disallow /store/index.php/
Disallow /cgi-bin/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php
Disallow /picture/view/*
Disallow /form_key*

Other Records

Field Value
crawl-delay 30
Back to top

Other Records

Field Value
sitemap https://www.buildersmart.in/sitemap.xml
Back to top

Comments

  • GENERAL SETTINGS
  • Enable robots.txt rules for all crawlers
  • Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
  • Set a custom crawl rate if you're experiencing traffic problems with your server.
  • Magento sitemap: uncomment and replace the URL to your Magento sitemap file
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • GENERAL MAGENTO SETTINGS
  • Do not crawl Magento admin page
  • Do not crawl common Magento technical folders
  • Do not crawl common Magento files
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • Do not crawl seach pages and not-SEO optimized catalog links
  • Disallow: /search?*
  • Disallow: /search/
  • Disallow: /catalog/product/view/
  • Video module fixes
  • CUSTOM ADDED BY GRAHAM
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
  • Disallow: */google-code-prettify*
Back to top