magicalgirlcafe.forumotion.com
robots.txt

Groups

mediapartners-google

mediapartners-google*

*

accoona
aipbot
aipbot*
aipbot/1.0
alexa
alexa bitlybot
alexibot
altavista intranet v2.0 avs eval search@freeit.com
altavista intranet v2.0 compaq altavista eval sveand@altavista.net
altavista intranet v2.0 evreka.com crawler@evreka.com
altavista v2.0b crawler@evreka.com
anonymous
apocalxexplorerbot
appie
aqua_products
argus/1.1
artabus
ask jeeves
asterias
atspider
attentio
av fetch 1.0
avsearch-3.0(altavista/avc)
aws cloud based
b2w
b2w/0.1
backdoorbot
backdoorbot/1.0
backlinkcrawler
becomebot
becomebot
bigbrother
biglotron (beta 2;gnu/linux)
bizinformation
black hole
black.hole
blackwidow
blowfish
blowfish/1.0
boardpulse
boitho.com-dc
bookmark search tool
bot/1.0
botalot
bot mailto:craftbot@yahoo.com
botrighthere
brandprotect
builtbottough
bullseye
bullseye/1.0
bunnyslippers
cazoodlebot
cegbfeieh
cfetch
cfetch/1.0
cheesebot
cherrypicker
cherrypicker /1.0
cherrypickerelite/1.0
cherrypickerse/1.0
chinaclaw
collage
cometrics-bot
complex_network_group
convera
converacrawler
converacrawler/0.2
converacrawler/0.9d
convera internet spider v6.x
converamultimediacrawler/0.1
copernic
copyrightcheck
cosmos
crescent
crescent internet toolpak http ole control v.1.0
crescent internet toolpak httpole control v.1.0
curl
custo
cydralspider
deepnet explorer
default.ida
digext
disco
discobot
discofinder
disco pump
disco pump 3.0
disco pump 3.1
disco pump 3.2
dittospyder
doc
dotbot
dotbot
dotbot/1.1
download demon
download demon/3.2.0.8
download demon/3.5.0.11
download ninja
download wonder
dsurf
dulance bot
dumbot
ecatch
ecatch/3.0
echo!
echo!/2.0
eirgrabber
elitesys entry
emailcollector
email extractor
emailsiphon
emailsmartz
emailwolf
enterprise_search
enterprise_search/1.0
erocrawler
es
esirover
e-societyrobot
exabot
exabot/2.0
exabot-images
express webpictures
express webpictures (www.express-soft.com)
extractorpro
eyenetie
fairad client
fairshare
fasterfox
fetch
findlinks
flaming attackbot
flamingo_searchengine
flashget
flashget webwasher 3.2
foobot
freefind
freewebmonitoring sitechecker/0.1
frontpage
frontpage [nc,or]
furlbot
gaisbot
gaisbot/3.0
getbot
getright
getright/2.11
getright/3.1
getright/3.2
getright/3.3
getright/3.3.3
getright/3.3.4
getright/4.0.0
getright/4.1.0
getright/4.1.1
getright/4.1.2
getright/4.2
getright/4.2b (portuguxeas)
getright/4.2c
getright/4.3
getright/4.5
getright/4.5a
getright/4.5b
getright/4.5b1
getright/4.5b2
getright/4.5b3
getright/4.5b6
getright/4.5b7
getright/4.5c
getright/4.5d
getright/4.5e
getright/5.0beta1
getright/5.0beta2
geturl
getweb!
gigabot
gigabot/3.0
go-ahead-got-it
go!zilla
go!zilla 3.3 (www.gozilla.com)
go!zilla 3.5 (www.gozilla.com)
go!zilla (www.gozilla.com)
grabnet
grafula
grub
grub-client
hackertarget.com
harvest
harvest/1.5
hatena antenna
havindex
heritrix
hloader
hmview
httplib
httrack
httrack
httrack 3.0
httrack 3.0x
httrack [nc,or]
humanlinks
ichiro
iconsurf
igentia
image collector
image stripper
image sucker
indy library
indy library [nc,or]
infonavirobot
infospiders
interget
internet explore
internet ninja
internet ninja 4.0
internet ninja 5.0
internet ninja 6.0
internetsupervision
irlbot
iron
iron33/1.0.2
jeeves
jennybot
jetbot
jetbot/1.0
jetcar
jobo
joc web spider
kalooga
kdd exploror
kenjin spider
kenjin.spider
keyword density
keyword.density
keyword density/0.9
larbin
larbin
larbin_2.6.2 (kabura@sushi.com)
larbin_2.6.2 kabura@sushi.com
larbin_2.6.2 (larbin2.6.2@unspecified.mail)
larbin_2.6.2 larbin2.6.2@unspecified.mail
larbin_2.6.2 larbin@correa.org
larbin_2.6.2 listonatccdotgatechdotedu
larbin_2.6.2 (listonatccdotgatechdotedu)
larbin_2.6.2 (vitalbox1@hotmail.com)
larbin_2.6.2 vitalbox1@hotmail.com
larbin (samualt9@bigfoot.com)
larbin samualt9@bigfoot.com
lbot
leechftp
lexibot
libweb/clshttp
libweb/clshttpdisallow: /
libwww
lightningdownload
linguee
linkedin
linkextractorpro
linknzbot
linknzbot*
linknzbot 2004
linkscan
linkscan/8.1a unix
linkscan/8.1a.unix
linkscan/8.1a unix disallow: /
linksmanager
linksmanager
linksmanager.com_bot
linkwalker
ljseek
lnspiderguy
looksmart
lwp
lwp*
lwp-trivial
lwp-trivial/1.34
magpie-crawler
mail sweeper
marketwirebot
mass downloader
mass downloader/2.2
mata hari
mata.hari
megaindex.ru
megaindex.ru/2.0
metagerbot
metauri
microsoft.url
microsoft url control
microsoft url control*
microsoft.url.control
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169
microsoft url control - 6.01.9782
midown tool
miixpc
miixpc/4.2
missigua locator
mister pix
mister.pix
mister pix ii 2.01
mister pix ii 2.02a
mister pix version.dll
mlbot
moget
moget/2.1
mozilla
mozilla
mozilla/2.0 (compatible; ask jeeves)
mozilla/3
mozilla/4
mozilla/4.0 (compatible; bullseye; windows 95)
mozilla/4.0 (compatible; msie 4.0; windows 2000)
mozilla/4.0 (compatible; msie 4.0; windows 95)
mozilla/4.0 (compatible; msie 4.0; windows 98)
mozilla/4.0 (compatible; msie 4.0; windows me)
mozilla/4.0 (compatible; msie 4.0; windows nt)
mozilla/4.0 (compatible; msie 4.0; windows xp)
mozilla/5
mrsputnik
msiecrawler
msrbot
ms search 4.0 robot
ms search 5.0 robot
munky
naver
naverbot
naverbot
naverbot-1.0
navroad
nearsite
netants
netants/1.10
netants/1.23
netants/1.24
netants/1.25
netattache
netattache light 1.1
netcraft web server survey
netmechanic
netspider
net vampire
net vampire/3.0
netzip
netzip-downloader
netzip-downloader/1.0.62 (win32; dec 7 1998)
netzip downloader 1.0 win32(nov 12 1998)
netzippy+(http://www.innerprise.net/usp-spider.asp)
netzippy+(http:/www.innerprise.net/usp-spider.asp)
nicerspro
nimblecrawler
npbot
npbot
npbot/3
nutch
nutch*
nutchcvs/0.06-dev
nutchcvs/0.7.1
nutchorg
obot
ocelli
octopus
offline explorer
offline.explorer
offline explorer/1.2
offline explorer/1.4
offline explorer/1.6
offline explorer/1.7
offline explorer/1.9
offline explorer/2.0
offline explorer/2.1
offline explorer/2.3
offline explorer/2.4
offline explorer/2.5
offline navigator
omniexplorer_bot
oneriot
openbot
openfind
openfind data gathere
openfind data gatherer
oracle ultra search
outfoxbot/0.5
pagegrabber
papa foto
pavuk
pbwf
pcbrowser
penthesilea
perman
pgbot
phpdig
pingdom gigrib (http://www.pingdom.com)
postrank
propowerbot
propowerbot/2.14
prowebwalker
psbot
psycheclone
psycheclone
python-urllib
quepasacreep
queryn metasearch
queryn.metasearch
radian6 comment reader
radian6 feedfetcher
radiation retriever
radiation retriever 1.1
rb2b-bot
realdownload
realdownload/4.0.0.40
realdownload/4.0.0.41
realdownload/4.0.0.42
reget
repomonkey
repomonkey bait & tackle/v1.01
repomonkey bait & tackle
repomonkey bait & tackle/v1.01
research-spider
rma
robozilla
roverbot
rufusbot
sbider
scooter/1.0
scooter/1.0 scooter@pa.dec.com
scooter/1.1 (custom)
scooter/2.0 g.r.a.b. v1.1.0
scooter/2.0 g.r.a.b. x2.0
scooter2_mercator_x-x.0
scooter-3.0.eu
scooter-3.0.fs
scooter-3.0.hd
scooter-3.0qi
scooter-3.0.vns
scooter-3.2
scooter-3.2.bt
scooter-3.2.dil
scooter-3.2.ex
scooter-3.2.jt
scooter-3.2.niv
scooter-3.2.sf0
scooter-3.2.snippet
scooter/3.3
scooter-3.3dev
scooter/3.3.qa.pczukor
scooter/3.3_sf
scooter/3.3.vscooter
scooter-ars-1.1
scooter-ars-1.1-ih
scooter_bh0-3.0.3
scooter_trk3-3.0.3
scooter-venus-3.0.vns
scooter-w3-1.0
scooter-w3.1.2
scrubby
searchdaimon.com-dc
searchpreview
semalt.com
seekbot
seekbot
seekbot/1.0
seoprofiler
shai'hulud
shim-crawler
shopwiki
shopwiki/1.0
sightupbot
sitebot
sitesnagger
slurp china
slysearch
smartdownload
smartdownload/1.2.76 (win32; apr 1 1999)
smartdownload/1.2.77 (win32; aug 17 1999)
smartdownload/1.2.77 (win32; feb 1 2000)
smartdownload/1.2.77 (win32; jun 19 2001)
snapbot
snappy
softlayer server
sogou web spider
sootle
sosospider
spankbot
spanner
spbot
speedy
spiderbot
sqworm
sqworm/2.9.85-beta (beta_release; 20011115-775; i686-pc-linux
ssearcher100
stanford
stanford comp sci
suggybot
superbot
superbot/2.6
superbot/3.0 (win32)
superbot/3.1 (win32)
superhttp
superhttp/1.0
surfbot
surveybot
suzuran
szukacz
szukacz/1.4
takeout
teleport
teleportpro
teleport pro
teleport pro/1.29
teleport pro/1.29.1590
teleport pro/1.29.1634
teleport pro/1.29.1718
teleport pro/1.29.1820
teleport pro/1.29.1847
telesoft
templeton
teoma
the intraformant
the.intraformant
thenomad
tighttwatbot
titan
tocrawl
tocrawl/urldispatcher
true_robot
true_robot/1.0
turingos
turnitinbot
turnitinbot/1.5
tweetmeme
twengabot
twiceler
url control
urldispatcher
url_spider_pro
urly warning
urly.warning
vci
vci webviewer vci webviewer win32
vobsub
voideye
vscooter
w3mir
watchdog/3.0
webauto
webauto/3.40 (win98; i)
webbandit
webbandit/3.50
webcapture
webcapture 2.0
webcatcher
webcopier
webcopier
webcopier v.2.2
webcopier v2.5
webcopier v2.6
webcopier v2.7a
webcopier v2.8
webcopier v3.0
webcopier v3.0.1
webcopier v3.2
webcopier v3.2a
webcopy
webcopy
webcrawl.net
webemailextrac
webemailextrac.*
webenhancer
webfetch
webfetch/2.1.0
webfetcher
webgo is
web image collector
web.image.collector
webleacher
webmasterworld extractor
webmasterworldforumbot
webmirror
webmirror
webreaper
web reaper
webreaper [info@webreaper.net]
webreaper v9.1 - www.otway.com/webreaper
webreaper v9.7 - www.webreaper.net
webreaper v9.8 - www.webreaper.net
webreaper vwebreaper v7.3 - www,otway.com/webreaper
webreaper [webreaper@otway.com]
websauger
websauger 1.20b
websauger 1.20j
websauger 1.20k
website extractor
website extractor
website extractor (http:/www.asona.org)
website quester
website.quester
website quester - www.asona.org
website quester - www.esalesbiz.com/extra/
webster pro
webster.pro
webstripper
webstripper/2.02
webstripper/2.03
webstripper/2.10
webstripper/2.12
webstripper/2.13
webstripper/2.15
webstripper/2.16
webstripper/2.19
web sucker
webvac
webvac
webvulncrawl
webvulnscan
webwalk
webwasher
webwhacker
webzip
webzip
webzip/2.75 (http://www.spidersoft.com)
webzip/2.75 (http:/www.spidersoft.com)
webzip/3.65 (http://www.spidersoft.com)
webzip/3.65 (http:/www.spidersoft.com)
webzip/3.80 (http://www.spidersoft.com)
webzip/3.80 (http:/www.spidersoft.com)
webzip/4.0
webzip/4.0 (http://www.spidersoft.com)
webzip/4.0 (http:/www.spidersoft.com)
webzip/4.1 (http://www.spidersoft.com)
webzip/4.1 (http:/www.spidersoft.com)
webzip/4.21
webzip/4.21 (http://www.spidersoft.com)
webzip/4.21 (http:/www.spidersoft.com)
webzip/5.0
webzip/5.0 (http://www.spidersoft.com)
webzip/5.0 (http:/www.spidersoft.com)
webzip/5.0 pr1 (http://www.spidersoft.com)
webzip/5.0 pr1 (http:/www.spidersoft.com)
wget
wget
wget
wget/1.10.2
wget/1.5.2
wget/1.5.3
wget/1.6
wget/1.7
wget/1.8
wget/1.8.1
wget/1.8.1+cvs
wget/1.8.2
wget/1.9-beta
whitevector crawler
whitevector+crawler
widow
wikiofeedbot
wikiwix-bot-3.0
willow
winhttrack
wise-guys
woozweb-monitoring
woriobot
www-collector
www-collector-e
wwwoffle
xaldon webspider
xaldon webspider 2.5.b3
xenu
xenu link sleuth
xenu's
xenu's link sleuth 1.1c
xget
yahoo-mmcrawler
yahooseeker/cafekelsa
yeti
yodaobot
yrspider
zao
zealbot
zeus
zeus 11389 webster pro v2.9 win32
zeus 11652 webster pro v2.9 win32
zeus 18018 webster pro v2.9 win32
zeus 26378 webster pro v2.9 win32
zeus 30747 webster pro v2.9 win32
zeus 32297 webster pro v2.9 win32
zeus 39206 webster pro v2.9 win32
zeus 41641 webster pro v2.9 win32
zeus 44238 webster pro v2.9 win32
zeus 51070 webster pro v2.9 win32
zeus 51674 webster pro v2.9 win32
zeus 51837 webster pro v2.9 win32
zeus 63567 webster pro v2.9 win32
zeus 6694 webster pro v2.9 win32
zeus 71129 webster pro v2.9 win32
zeus 82016 webster pro v2.9 win32
zeus 82900 webster pro v2.9 win32
zeus 84842 webster pro v2.9 win32
zeus 90872 webster pro v2.9 win32
zeus 94934 webster pro v2.9 win32
zeus 95245 webster pro v2.9 win32
zeus 95351 webster pro v2.9 win32
zeus 97371 webster pro v2.9 win32
zeus link scout
zyborg

ahrefsbot
semrushbot
sogou web spider
sogou spider
mj12bot
mj12bot/v1.4.3

No rules defined. All paths allowed.

Other Records

Other Records

Warnings

• 2 invalid lines.
• `noindex` is not a known field.