btech.com
robots.txt

Robots Exclusion Standard data for btech.com

Resource Scan

Scan Details

Site Domain btech.com
Base Domain btech.com
Scan Status Ok
Last Scan2024-06-03T16:23:04+00:00
Next Scan 2024-06-17T16:23:04+00:00

Last Scan

Scanned2024-06-03T16:23:04+00:00
URL https://btech.com/robots.txt
Domain IPs 104.26.4.24, 104.26.5.24, 172.67.71.23, 2606:4700:20::681a:418, 2606:4700:20::681a:518, 2606:4700:20::ac43:4717
Response IP 104.26.5.24
Found Yes
Hash 6890320df3105e8243b64f4a6638b4e9660b53dfa2f25d7fbe71ef59be5c3934
SimHash 6243ef9acbf0

Groups

*

Rule Path
Allow /arblog/
Allow /blog/
Allow /media/catalog/
Disallow /index.php/
Disallow /*?
Disallow /en/checkout/
Disallow /ar/checkout/
Disallow /app/
Disallow /lib/
Disallow /*.php$
Disallow /pkginfo/
Disallow /report/
Disallow /var/
Disallow /ar/customer/
Disallow /en/customer/
Disallow /sendfriend/
Disallow /review/
Disallow /*SID%3D
Disallow /en/catalog/brand/view/id/
Disallow /ar/catalog/brand/view/id/

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

queryn metasearch

Rule Path
Disallow /

true_robot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

Comments

  • Block access to specific groups of pages
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Block EtaoSpider as ecommerence engine
  • Unsafe robots to disallow