pret-simulation.be
robots.txt

Groups

*

abonti

abonti/0.92

aboutthedomain

add catalog

add catalog/2.1

advbot

advbot/2.0

affectv robot v1.0/nutch-1.6

ahrefsbot

aitellu

apache-httpclient/4.1.1 (java 1.5)

baiduspider

baidu spider

battleztar bazinga

battleztar bazinga/0.01

betabot

blackboard safeassign

blazer 1.0

blexbot

blexbot/1.0

blp_bbot

blp_bbot/0.1

boia-accessibility-agent

boia-accessibility-agent/pr 1.0

bot for jce

bot/0.1 (bot for jce)

bubing

ca-crawler

ca-crawler/1.0

cakephp

camelhttpstream/1.0 evolution/3.2.2

cb/nutch-1.7

ccbot

ccbot/2.0

cliqzbot

cliqzbot/0.1

cms crawler

coccoc

copyscape

crawler4j

crazywebcrawler

crazywebcrawler-spider

curl/7.15.3

curl/7.26.0

curl/7.32.0

curl/7.37.0

curl/php 5.3.29

cuwhois

cuwhois/1.0

dispatch/0.11.0

domainappender

domainappender /1.0

domainsigmacrawler

domainsigmacrawler/0.1

dotbot

ecommercebot

episerver link checker

euripbot

euripbot/2.0

exabot

findxbot

findxbot/1.0

flamingo_searchengine

flipboard robot

getrequest

getrequest/1.0

gigablastopensource

gigablastopensource/1.0

gluten free crawler

gluten free crawler/1.0

go 1.1 package http

haiula

haiula/1.4

haosouspider

heritrix

heritrix/1.14.4

heritrix/3.1.1

hivemind

http://git.io/tl_s2w

httpunit/1.x

hubspot links crawler 1.0

hypercrawl

hypercrawl/0.2

ichiro robot

image.coccoc/1.0

indy library

integrity/5

internaetboten

internaetboten/0.99

jakarta

jakarta commons

jakarta commons-httpclient

jakarta commons-httpclient/3.0.1

jakarta commons-httpclient/3.0-rcx

jakarta commons-httpclient/3.1

jamesbot

james bot

java/1.4.1_04

java/1.6.0_04

java/1.6.0_10-rc2

java/1.6.0_15

java/1.6.0_24

java/1.6.0_26

java/1.6.0_32

java/1.7.0_09

java/1.7.0_45

java/1.7.0_55

java/1.7.0_60

java/1.7.0_60-ea

java/1.7.0_65

java/1.7.0_67

java/1.7.0_71

java/1.8.0_20

java/1.8.0_25

java/1.8.0_31

jetbrains

jetbrains 5.0

kraken

kraken/0.1

kraken/nutch-2.2.1

larbin

libcurl

libcurl/7.15.3

libcurl/7.19.7

libcurl/7.21.0

libreoffice

link/1.0

linkdex

linkdexbot

linkdexbot/2.0

linkdexbot/2.1

linkdex.com/v2.0

linkdex.com/v2.1

linkdexbot-mobile/2.1

linkpadbot

linkpadbot/1.06

lipperhey

lipperhey seo service

lipperhey-kaus-australis

lipperhey-kaus-australis/5.0

listicka

lssrocketcrawler

lssrocketcrawler/1.0

lssrocketcrawler/1.0 lightspeedsystems

ltx71

lwnutch/nutch-1.4

magpie-crawler

mail.ru

mail.ru_bot

mail.ru_bot/2.0

mail.ru_bot/fast/2.0

maxpointcrawler

maxpointcrawler/nutch-1.1

md5sum

md5sum\x22

meanpathbot

megaindex.ru

megaindex.ru/2.0

mixbot

mj12bot

my nutch spider/nutch-1.9

nerdybot

netseer

netvibes

ning/1.0

node.io

node.js

obot/2.3.1

omgilibot

omgilibot/0.4

openstat

openstat/0.1

optimizationcrawler

optimizationcrawler/0.2

pageanalyzer

pageanalyzer/1.1

pagesinventory

pagespeedbot

pagespeed/1.1

pagespeed/1.1 fetcher

pathdefender

pathdefender/1.0

perl lwp

plukkie

plukkie/1.5

pogs/2.0

privacy_enfaq.jsp

prlog

prlog/1.0

psbot

publiclibraryarchive.org

publiclibraryarchive.org/1.0

pycurl

pycurl/7.24.0

python-urllib

python-urllib/1.17

python-urllib/2.5

python-urllib/2.6

python-urllib/2.7

r6_feedfetcher

r6_commentreader

riddler

rogerbot

rogerbot/1.0

rome client

rootlink

ru_bot/2.0

ruby

scrapy

scrapy/0.24.4

screaming frog seo spider

screaming frog seo spider/2,55

semrushbot

semrushbot-sa

seznambot

showyoubot

sheerboredom.experimental.robot

sheerboredom.experimental.robot/0.2

sistrix

sistrix crawler

siteexplorer

siteexplorer/1.0b

skimbot

skimbot/1.0

smtbot

smtbot/1.0

sogou pic spider

sogou pic spider/3.0

sogou spider

sogou web spider

spbot

spbot/4.4.2

spiderbot/nutch-1.7

ssg/3.0

steeler

steeler/3.5

studiofaca

studiofaca search

sukibot

sukibot_heritrix

sukibot_heritrix/3.1.1

surveybot

synthesio crawler release monalisa

trendictionbot

truebot

truebot/1.0

typhoeus

uipbot/1.0

umbot-ln

umbot-ln/1.0

updown_tester

urlchecker

webcookies

webcookies/1.0

webcopier vx.xa

web corpus crawler

web-sniffer

web-sniffer/1.1.0

wegtam

wegtam crawler

whatweb

whatweb/0.4.8-dev

worldbrewbot

worldbrewbot/2.1

worldwebheritage.org

worldwebheritage.org/1.0

www.deadlinkchecker.com

v1.0/1.2

vbseo

vericitecrawler

vericitecrawler/nutch-1.9

voila robot

voltron

vorboss web crawler

vorboss web crawler/nutch-2.3

xenu link sleuth

xenu's link sleuth

xovibot

xovibot/2.0

yandex

yandex robot

yetibot

yisouspider

zzabmbot

zzabmbot/1.0

boitho.com-dcuser-agent: cazoodlebotuser-agent: converacrawleruser-agent: discobotuser-agent: dotbotuser-agent: fasterfoxuser-agent: gigabotuser-agent: heritrixuser-agent: huaweisymantecspideruser-agent: ia_archiveruser-agent: ichirouser-agent: irlbotuser-agent: kaloogauser-agent: lexxebotuser-agent: mj12botuser-agent: mlbotuser-agent: msiecrawleruser-agent: naverbotuser-agent: nutchuser-agent: ocelliuser-agent: penthesileauser-agent: phpdiguser-agent: scoutjetuser-agent: searchdaimon.com-dcuser-agent: snapbotuser-agent: turnitinbotuser-agent: twiceleruser-agent: woozweb-monitoringuser-agent: woriobotuser-agent: yacybituser-agent: yetiuser-agent: yodaobotuser-agent: zyborguser-agent: alexibotuser-agent: apocalxexplorerbotuser-agent: asteriasuser-agent: backdoorbot/1.0user-agent: black holeuser-agent: blowfish/1.0user-agent: botalotuser-agent: builtbottoughuser-agent: bullseye/1.0user-agent: bunnyslippersuser-agent: cegbfeiehuser-agent: cheesebotuser-agent: cherrypickeruser-agent: cherrypickerelite/1.0user-agent: cherrypickerse/1.0user-agent: copyrightcheckuser-agent: cosmosuser-agent: crescentuser-agent: crescent internet toolpak http ole control v.1.0user-agent: disco pump 3.1user-agent: dittospyderuser-agent: emailcollectoruser-agent: emailsiphonuser-agent: emailwolfuser-agent: erocrawleruser-agent: extractorprouser-agent: foobotuser-agent: harvest/1.5user-agent: hloaderuser-agent: httplibuser-agent: httrackuser-agent: httrack 3.0user-agent: humanlinksuser-agent: infonavirobotuser-agent: jennybotuser-agent: kenjin spideruser-agent: lexibotuser-agent: libweb/clshttpuser-agent: linkextractorprouser-agent: linkscan/8.1a unixuser-agent: linkwalkeruser-agent: lwp-trivialuser-agent: lwp-trivial/1.34user-agent: mata hariuser-agent: microsoft url control - 5.01.4511user-agent: microsoft url control - 6.00.8169user-agent: miixpcuser-agent: miixpc/4.2user-agent: mister pixuser-agent: mogetuser-agent: moget/2.1user-agent: netantsuser-agent: netattacheuser-agent: netattache light 1.1user-agent: netmechanicuser-agent: nicersprouser-agent: offline exploreruser-agent: openfinduser-agent: openfind data gathereuser-agent: propowerbot/2.14user-agent: prowebwalkeruser-agent: psbotuser-agent: quepasacreep user-agent: queryn metasearchuser-agent: repomonkeyuser-agent: repomonkey bait & tackle/v1.01user-agent: rmauser-agent: sitesnaggeruser-agent: spankbotuser-agent: spanneruser-agent: superbotuser-agent: superbot/2.6user-agent: suzuranuser-agent: szukacz/1.4user-agent: teleportuser-agent: telesoftuser-agent: the intraformantuser-agent: thenomaduser-agent: tighttwatbotuser-agent: titanuser-agent: tocrawl/urldispatcheruser-agent: true_robotuser-agent: true_robot/1.0user-agent: turingosuser-agent: urly warninguser-agent: vciuser-agent: vci webviewer vci webviewer win32user-agent: web image collectoruser-agent: webautouser-agent: webbandituser-agent: webbandit/3.50user-agent: webcopieruser-agent: webcopyuser-agent: webenhanceruser-agent: webmasterworldforumbotuser-agent: webmirroruser-agent: webreaperuser-agent: websaugeruser-agent: website extractoruser-agent: website questeruser-agent: webster prouser-agent: webstripperuser-agent: webstripper/2.02user-agent: webzipuser-agent: webzip/4.0user-agent: wgetuser-agent: wget/1.5.3user-agent: wget/1.6user-agent: winhttrackuser-agent: zeususer-agent: zeus 32297 webster pro v2.9 win32disallow: /

No rules defined. All paths allowed.

Warnings

• 14 invalid lines.