Log In Sign Up

btxonline.com
robots.txt

Robots Exclusion Standard data for btxonline.com

Resource Scan

Scan Details

Site Domain btxonline.com
Base Domain btxonline.com
Scan Status Ok
Last Scan2025-07-27T20:13:00+00:00
Next Scan 2025-08-10T20:13:00+00:00

Last Scan

Scanned2025-07-27T20:13:00+00:00
URL https://btxonline.com/robots.txt
Domain IPs 52.84.45.18, 52.84.45.2, 52.84.45.38, 52.84.45.67
Response IP 18.165.72.17
Found Yes
Hash ba7a547081903f067e089e4475fa54347f072ccaca4ae9667b03166c9f652c70
SimHash 61574f174dd3

Groups

*

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap http://www.btxonline.com/sitemap.xml
Back to top

Comments

  • Robots.txt file for www.btxonline.com
  • Enable robots.txt rules for all crawlers
  • Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
  • Set a custom crawl rate if you're experiencing traffic problems with your server.
  • Crawl-delay: 30
  • Magento sitemap: uncomment and replace the URL to your Magento sitemap file
Back to top