farmer.pl
robots.txt

Groups

*

bingbot

No rules defined. All paths allowed.

Other Records

yandex

aboundex
accelobot
add\ catalog
ahrefsbot
aihitbot
alexibot
aqua_products
askjeeves
asterias
awcheckbot
b2w/0.1
backdoorbot/1.0
backlinkcrawler
baiduspider
becomebot
blexbot
blowfish/1.0
bookmark search tool
botalot
brandwatch.net
builtbottough
bullseye/1.0
bunnyslippers
butterfly
catchbot
charlotte
cheesebot
cherrypicker
cherrypickerelite/1.0
cherrypickerse/1.0
clipish
cliqzbot
comodo
comodo-certificates-spider
compspybot
copernic
copyrightcheck
cosmos
crawler
crescent
crescent internet toolpak http ole control v.1.0
curious
curl
dataprovider\.com
dinoping
discoverybot
dittospyder
domaincrawler
domaincrawler
dotbot
dotnetdotcom
dow\ jones\ searchbot
dumbot
easouspider
emailcollector
emailsiphon
emailwolf
enterprise_search
enterprise_search/1.0
erocrawler
es
exabot
extractorpro
ezinearticleslinkscanner
ezooms
fairad client
flaming attackbot
foobot
freefind
ftrf\:\ friendly
gaisbot
getright/4.2
gigabot
grub
grub-client
harvest/1.5
hatena antenna
hloader
http://www.searchengineworld.com bot
http://www.webmasterworld.com bot
http_request
http_request2
httplib
humanlinks
ia_archiver
ia_archiver
ia_archiver/1.6
indy\ library
infonavirobot
ip\-web\-crawler\.com
iron33/1.0.2
jakarta\ commons-httpclient
jeeves
jennybot
jetbot
jetbot/1.0
jikespider
kenjin spider
keyword density/0.9
larbin
lexibot
libweb/clshttp
libwww-perl
lindex\.com
linguee
linkdex\.com
linkdexbot
linkextractorpro
linkscan/8.1a unix
linkwalker
lipperhey
lnspiderguy
looksmart
ltbot
lwp-trivial
lwp-trivial/1.34
magpie\-crawler
mata hari
microsoft url control
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169
miixpc
miixpc/4.2
mister pix
mj12bot
moget
moget/2.1
msie\ or\ firefox\ mutant
msiecrawler
naver
ncbot
netants
netcraftsurveyagent
netestate\ ne\ crawler
netmechanic
netseer
nextgensearchbot
nicerspro
nutch
nutch
ocelli
offline explorer
omniexplorer_bot
openbot
openfind
openfind
openfind data gathere
openwebindex
oracle ultra search
pagesinventory
pear
peoplepal
perman
procogseobot
propowerbot/2.14
prowebwalker
proximic
psbot
purebot
queryn metasearch
queryseekerspider
radiation retriever 1.1
repomonkey
repomonkey bait & tackle/v1.01
riddler
rma
rojerbot
rytebot
scooter
scoutjet
scrapy
screenerbot
searchmetrics
searchpreview
semrushbot
sentibot
seo-crawling
seoengworldbot
seokicks-robot
shopwiki
sistrix
sitebot
sitesnagger
snoopy
socialsearcher
sogou
sogou
solomonobot
sootle
sosospider
spankbot
spanner
spbot
speedy
stanford
stanford comp sci
surveybot
suzuran
szukacz/1.4
szukacz/1.4
teleport
teleportpro
telesoft
teoma
the intraformant
the\ incutio\ xml-rpc\ php\ library
thenomad
tocrawl/urldispatcher
true_robot
true_robot/1.0
turingos
turnitinbot
ucrawler
url control
url_spider_pro
urly warning
vci
vci webviewer vci webviewer win32
visaduhoc\.info
wbsearchbot
web image collector
webauto
webbandit
webbandit/3.50
webcapture
webcopier
webenhancer
webindetail\.com
webmasterworld extractor
webmasterworldforumbot
websauger
website quester
websitetheweb\.com
webster pro
webstripper
webvac
webzip
webzip/4.0
wget
wget/1.5.3
wget/1.6
wotbot
www\.integromedb\.org
www-collector-e
xpymep\.exe
yamanalab-robot
yisouspider
yodaobot
youdaobot
zend_http_client
zeus
zeus 32297 webster pro v2.9 win32
zeus link scout
zmeu
zumbot

ai2bot
ai2bot-dolma
amazonbot
anthropic-ai
applebot
applebot-extended
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
diffbot
facebookbot
friendlycrawler
google-extended
googleother
googleother-image
googleother-video
gptbot
iaskspider/2.0
icc-crawler
imagesiftbot
img2dataset
meta-externalagent
meta-externalfetcher
moodlebot
newsnow
oai-searchbot
omgili
omgilibot
perplexitybot
petalbot
piplbot
scrapy
timpibot
velenpublicwebcrawler
webzio-extended
youbot

Other Records

Comments

• SearchEngines
• 3rdParties
• AI/LLMs
• Sitemaps

Warnings

• 4 invalid lines.