Log In Sign Up

bsti.be
robots.txt

Robots Exclusion Standard data for bsti.be

Resource Scan

Scan Details

Site Domain bsti.be
Base Domain bsti.be
Scan Status Ok
Last Scan2025-07-26T08:22:38+00:00
Next Scan 2025-08-25T08:22:38+00:00

Last Scan

Scanned2025-07-26T08:22:38+00:00
URL https://bsti.be/robots.txt
Domain IPs 2a02:2350:5:10b:4d:8ddc:66ce:6061, 46.30.213.73
Response IP 46.30.213.73
Found Yes
Hash 89dd2a4d33474673db09070b04fc4a4cfd181ef588ef13caa93ce575a6e83f31
SimHash 9d945da64f52

Groups

*

Rule Path
Disallow /cgi-bin/

*

Rule Path
Disallow /mrbs/

*

Rule Path
Disallow /claro/

atspider

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

dsurf

Rule Path
Disallow /

elitesys entry

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

mail sweeper

Rule Path
Disallow /

munky

Rule Path
Disallow /

roverbot

Rule Path
Disallow /

webemailextrac

Rule Path
Disallow /

xget

Rule Path
Disallow /

wget

Rule Path
Disallow /

webwalk

Rule Path
Disallow /

webvac

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webmirror

Rule Path
Disallow /

webfetcher

Rule Path
Disallow /

webcopy

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webcatcher

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

w3mir

Rule Path
Disallow /

vobsub

Rule Path
Disallow /

templeton

Rule Path
Disallow /

ssearcher100

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

shai'hulud

Rule Path
Disallow /

pbwf

Rule Path
Disallow /

lightningdownload

Rule Path
Disallow /

kdd exploror

Rule Path
Disallow /

jeeves

Rule Path
Disallow /

internet explore

Rule Path
Disallow /

infospiders

Rule Path
Disallow /

havindex

Rule Path
Disallow /

geturl

Rule Path
Disallow /

getbot

Rule Path
Disallow /

esirover

Rule Path
Disallow /

download wonder

Rule Path
Disallow /

collage

Rule Path
Disallow /

mozilla/2.0 (compatible; ms frontpage 4.0)

Rule Path
Disallow /
Back to top

Comments

  • Robots.txt for Euregio.Net Hosted Web Site
  • Disallow indexing of specific file extensions
  • Disallow Collectors and Spam
  • Disallow Offline Browsers
  • Disallow: /
  • User-agent: httrack
Back to top