helaldunyamarketleri.com robots.txt

googlebot

Rule	Path
Allow	/images/thumbnails/
Disallow	/ProductTab/

Rule

Path

Allow

/images/thumbnails/

Disallow

/ProductTab/

googlebot-image

Rule	Path
Allow	/images/thumbnails/

Rule

Path

Allow

/images/thumbnails/

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

sp_auditbot

Rule	Path
Allow	/

Rule

Path

Allow

/

riddler

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blp_bbot/0.1

Rule	Path
Disallow	/

Rule

Path

Disallow

/

user-agent

Rule	Path
Disallow	/

Rule

Path

Disallow

/

compatible ;

Rule	Path
Disallow	/

Rule

Path

Disallow

/

mozilla

Rule	Path
Disallow	/

Rule

Path

Disallow

/

libwww

Rule	Path
Disallow	/

Rule

Path

Disallow

/

lwp-trivial

Rule	Path
Disallow	/

Rule

Path

Disallow

/

curl

Rule	Path
Disallow	/

Rule

Path

Disallow

/

php/

Rule	Path
Disallow	/

Rule

Path

Disallow

/

urllib

Rule	Path
Disallow	/

Rule

Path

Disallow

/

gt::www

Rule	Path
Disallow	/

Rule

Path

Disallow

/

snoopy

Rule	Path
Disallow	/

Rule

Path

Disallow

/

mfc_tear_sample

Rule	Path
Disallow	/

Rule

Path

Disallow

/

http::lite

Rule	Path
Disallow	/

Rule

Path

Disallow

/

phpcrawl

Rule	Path
Disallow	/

Rule

Path

Disallow

/

uri::fetch

Rule	Path
Disallow	/

Rule

Path

Disallow

/

zend_http_client

Rule	Path
Disallow	/

Rule

Path

Disallow

/

http client

Rule	Path
Disallow	/

Rule

Path

Disallow

/

pecl::http

Rule	Path
Disallow	/

Rule

Path

Disallow

/

panscient.com

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ibm evv

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bork-edition

Rule	Path
Disallow	/

Rule

Path

Disallow

/

fetch api request

Rule	Path
Disallow	/

Rule

Path

Disallow

/

wep search

Rule	Path
Disallow	/

Rule

Path

Disallow

/

wells search ii

Rule	Path
Disallow	/

Rule

Path

Disallow

/

missigua locator

Rule	Path
Disallow	/

Rule

Path

Disallow

/

indy library

Rule	Path
Disallow	/

Rule

Path

Disallow

/

java/1.4.1_04

Rule	Path
Disallow	/

Rule

Path

Disallow

/

mj12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)

Rule	Path
Disallow	/

Rule

Path

Disallow

/

java/1.6.0_04

Rule	Path
Disallow	/

Rule

Path

Disallow

/

java/1.5.0_06

Rule	Path
Disallow	/

Rule

Path

Disallow

/

java/1.6.0_11

Rule	Path
Disallow	/

Rule

Path

Disallow

/

java/1.6.0_07

Rule	Path
Disallow	/

Rule

Path

Disallow

/

java/1.6.0_13

Rule	Path
Disallow	/

Rule

Path

Disallow

/

isc systems irc search 2.1

Rule	Path
Disallow	/

Rule

Path

Disallow

/

mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aisearchbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

alexibot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

amazoncloudservices

Rule	Path
Disallow	/

Rule

Path

Disallow

/

apexoo spider 1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aqua_products

Rule	Path
Disallow	/

Rule

Path

Disallow

/

arachnoidea

Rule	Path
Disallow	/

Rule

Path

Disallow

/

architextspider

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aspsimply

Rule	Path
Disallow	/

Rule

Path

Disallow

/

asterias

Rule

Path

Disallow

/

attributorcorporationdmcabot

Rule

Path

Disallow

/

avant+browser+

Rule

Path

Disallow

/

axonize-bot

Rule

Path

Disallow

/

b2w/0.1

Rule

Path

Disallow

/

backdoorbot/1.0

Rule

Path

Disallow

/

baiduspider

Rule

Path

Disallow

/

baiduspider+(+http://www.baidu.com/search/spider_jp.html)

Rule

Path

Disallow

/

becomebot

Rule

Path

Disallow

/

becomebot/3.0

Rule

Path

Disallow

/

bitacle

Rule

Path

Disallow

/

bitacle bot

Rule

Path

Disallow

/

bitacle bot/1.1

Rule

Path

Disallow

/

bitacle*

Rule

Path

Disallow

/

black hole

Rule

Path

Disallow

/

blowfish/1.0

Rule

Path

Disallow

/

bobcrawl

Rule

Path

Disallow

/

boitho

Rule

Path

Disallow

/

boitho.com

Rule

Path

Disallow

/

bookmark search tool

Rule

Path

Disallow

/

bot/1.0

Rule

Path

Disallow

/

bot/1.0+(bot;+http://;+bot@bot.bot)

Rule

Path

Disallow

/

botalot

Rule

Path

Disallow

/

botmobi

Rule

Path

Disallow

/

bsalsa

Rule

Path

Disallow

/

builtbottough

Rule

Path

Disallow

/

bullseye/1.0

Rule

Path

Disallow

/

bunnyslippers

Rule

Path

Disallow

/

butterfly

Rule

Path

Disallow

/

cazoodle-bot

Rule

Path

Disallow

/

cazoodlebot

Rule

Path

Disallow

/

cegbfeieh

Rule

Path

Disallow

/

charlotte

Rule

Path

Disallow

/

cheesebot

Rule

Path

Disallow

/

cherrypicker

Rule

Path

Disallow

/

cherrypickerelite/1.0

Rule

Path

Disallow

/

cherrypickerse/1.0

Rule

Path

Disallow

/

classroombot

Rule

Path

Disallow

/

converacrawler/0.9d

Rule

Path

Disallow

/

converacrawler/0.9e

Rule

Path

Disallow

/

copernic

Rule

Path

Disallow

/

copyrightcheck

Rule

Path

Disallow

/

cosmos

Rule

Path

Disallow

/

crawl13.exabot.com

Rule

Path

Disallow

/

crawl6.exabot.com

Rule

Path

Disallow

/

crescent

Rule

Path

Disallow

/

crescent internet toolpak http ole control v.1.0

Rule

Path

Disallow

/

cyberpatrol

Rule

Path

Disallow

/

cyberpatrolcrawler

Rule

Path

Disallow

/

datacha0s/2.0

Rule

Path

Disallow

/

dblbot

Rule

Path

Disallow

/

dcbot

Rule

Path

Disallow

/

del.icio.us

Rule

Path

Disallow

/

del.icio.us-thumbnails/1.0

Rule

Path

Disallow

/

depspid

Rule

Path

Disallow

/

discobot

Rule

Path

Disallow

/

dittospyder

Rule

Path

Disallow

/

docomo

Rule

Path

Disallow

/

dotbot

Rule

Path

Disallow

/

dotbot

Rule

Path

Disallow

/

dtaagent

Rule

Path

Disallow

/

eagle

Rule

Path

Disallow

/

ee:

Rule

Path

Disallow

/

emailcollector

Rule

Path

Disallow

/

emailsiphon

Rule

Path

Disallow

/

emailwolf

Rule

Path

Disallow

/

envolk

Rule

Path

Disallow

/

erocrawler

Rule

Path

Disallow

/

eudora

Rule

Path

Disallow

/

everest-vulcan

Rule

Path

Disallow

/

exabot

Rule

Path

Disallow

/

exabot

Rule

Path

Disallow

/

exabot-images/1.0

Rule

Path

Disallow

/

exabot-images/3.0

Rule

Path

Disallow

/

exabot/2.0

Rule

Path

Disallow

/

exabot/3.0

Rule

Path

Disallow

/

exooba

Rule

Path

Disallow

/

extractorpro

Rule

Path

Disallow

/

fairad client

Rule

Path

Disallow

/

fairshare

Rule

Path

Disallow

/

fast

Rule

Path

Disallow

/

fast enterprise crawler

Rule

Path

Disallow

/

fast metaweb crawler

Rule

Path

Disallow

/

fast-webcrawler

Rule

Path

Disallow

/

fast-webcrawler

Rule

Path

Disallow

/

findlinks/1.1.1-a5

Rule

Path

Disallow

/

findlinks/1.1.3-beta9

Rule

Path

Disallow

/

flaming attackbot

Rule

Path

Disallow

/

flatlandbot

Rule

Path

Disallow

/

followsitebot

Rule

Path

Disallow

/

foobot

Rule

Path

Disallow

/

funwebproducts

Rule

Path

Disallow

/

funwebproducts f39-2350

Rule

Path

Disallow

/

fyberspider

Rule

Path

Disallow

/

gaisbot

Rule

Path

Disallow

/

geniebot

Rule

Path

Disallow

/

geniebot (wgao@genieknows.com)

Rule

Path

Disallow

/

geniebot+(wgao@genieknows.com)

Rule

Path

Disallow

/

geniebot+wgao@genieknows.com

Rule

Path

Disallow

/

getright/4.2

Rule

Path

Disallow

/

gigabot

Rule

Path

Disallow

/

gigabot*

Rule

Path

Disallow

/

gigabot/2.0

Rule

Path

Disallow

/

gingercrawler

Rule

Path

Disallow

/

grub

Rule

Path

Disallow

/

grub-client

Rule

Path

Disallow

/

gsa-crawler

Rule

Path

Disallow

/

gulliver

Rule

Path

Disallow

/

harvest/1.5

Rule

Path

Disallow

/

henrythemiragorobot

Rule

Path

Disallow

/

heritrix

Rule

Path

Disallow

/

heritrix/1.12.1 +http://www.page-store.com)

Rule

Path

Disallow

/

hloader

Rule

Path

Disallow

/

hmse_robot

Rule

Path

Disallow

/

hornysexsearch

Rule

Path

Disallow

/

http://www.almaden.ibm.com/cs/crawler

Rule

Path

Disallow

/

http://www.cuill.com/twiceler/robot.html

Rule

Path

Disallow

/

http/1.1 zibber-v0.1(www.zibb.com/crawler/)

Rule

Path

Disallow

/

httplib

Rule

Path

Disallow

/

humanlinks

Rule

Path

Disallow

/

ia_archive

Rule

Path

Disallow

/

ia_archiver-web.archive.org

Rule

Path

Disallow

/

ia_archiver/1.6

Rule

Path

Disallow

/

ibm

Rule

Path

Disallow

/

ibm_planetwide

Rule

Path

Disallow

/

ichiro/2.0

Rule

Path

Disallow

/

ichiro/2.0 (ichiro@nttr.co.jp)

Rule

Path

Disallow

/

ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html)

Rule

Path

Disallow

/

iconsurf

Rule

Path

Disallow

/

infonavirobot

Rule

Path

Disallow

/

infopath

Rule

Path

Disallow

/

infopath.1

Rule

Path

Disallow

/

infopath.2

Rule

Path

Disallow

/

integratelecom

Rule

Path

Disallow

/

internetseer

Rule

Path

Disallow

/

internetseer.com

Rule

Path

Disallow

/

ips-agent

Rule

Path

Disallow

/

irlbot

Rule

Path

Disallow

/

irlbot/3.0

Rule

Path

Disallow

/

irlbot/3.0 (compatible; msie 6.0; http://irl.cs.tamu.edu/crawler)

Rule

Path

Disallow

/

iron33/1.0.2

Rule

Path

Disallow

/

isidorus

Rule

Path

Disallow

/

jakarta commons-httpclient

Rule

Path

Disallow

/

java

Rule

Path

Disallow

/

java bot 2.0

Rule

Path

Disallow

/

java/1.5.0_05

Rule

Path

Disallow

/

java/1.5.0_10

Rule

Path

Disallow

/

javabot

Rule

Path

Disallow

/

javabot 2.0

Rule

Path

Disallow

/

jennybot

Rule

Path

Disallow

/

jyxobot

Rule

Path

Disallow

/

kalooga

Rule

Path

Disallow

/

kaloogabot

Rule

Path

Disallow

/

kenjin spider

Rule

Path

Disallow

/

keyword density/0.9

Rule

Path

Disallow

/

kyluka

Rule

Path

Disallow

/

labot

Rule

Path

Disallow

/

larbin

Rule

Path

Disallow

/

larbin_2.6.1

Rule

Path

Disallow

/

larbin_2.6.1+(larbin2.6.1@unspecified.mail)

Rule

Path

Disallow

/

laycat

Rule

Path

Disallow

/

lexibot

Rule

Path

Disallow

/

lexibotweb image collector

Rule

Path

Disallow

/

libcurl-agent

Rule

Path

Disallow

/

libweb/clshttp

Rule

Path

Disallow

/

libwww-perl

Rule

Path

Disallow

/

libwww-perl/5.65

Rule

Path

Disallow

/

linkaider

Rule

Path

Disallow

/

linkextractorpro

Rule

Path

Disallow

/

linkscan/8.1a unix

Rule

Path

Disallow

/

linkwalker

Rule

Path

Disallow

/

linkwalker/2.0

Rule

Path

Disallow

/

linkwalker/2.0 - www.seventwentyfour.com

Rule

Path

Disallow

/

lmqueuebot/0.2

Rule

Path

Disallow

/

lnspiderguy

Rule

Path

Disallow

/

localbot

Rule

Path

Disallow

/

localcombot

Rule

Path

Disallow

/

looksmart

Rule

Path

Disallow

/

lwp-trivial/1.34

Rule

Path

Disallow

/

mail.ru

Rule

Path

Disallow

/

mantraagent

Rule

Path

Disallow

/

mata hari

Rule

Path

Disallow

/

mediapartners-google*

Rule

Path

Disallow

/

microsoft url control

Rule

Path

Disallow

/

microsoft url control - 5.01.4511

Rule

Path

Disallow

/

microsoft url control - 6.00.8169

Rule

Path

Disallow

/

miixpc

Rule

Path

Disallow

/

miixpc/4.2

Rule

Path

Disallow

/

mister pix

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

mj12bot/v1.2.0

Rule

Path

Disallow

/

mj12bot/v1.2.0 (http://majestic12.co.uk/bot.php? )

Rule

Path

Disallow

/

mlbot

Rule

Path

Disallow

/

modified gmmbotrighthere

Rule

Path

Disallow

/

moget

Rule

Path

Disallow

/

moget/2.1

Rule

Path

Disallow

/

moget/2.1 (moget@goo.ne.jp)

Rule

Path

Disallow

/

moozilla

Rule

Path

Disallow

/

mozilla/2.0 (compatible; ask jeeves/teoma)

Rule

Path

Disallow

/

mozilla/3

Rule

Path

Disallow

/

mozilla/4

Rule

Path

Disallow

/

mozilla/4.0

Rule

Path

Disallow

/

mozilla/4.0 (compatible; bullseye; windows 95)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows 2000)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows 9

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows 95)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows 98)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows nt)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows xp)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 5.0; windows 95) voilabot beta 1.2

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 5.05; windows nt 3.51)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 5.05; windows nt 4.0)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 5.05; windows nt 5.0)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 6.0; windows nt 5.0)

Rule

Path

Disallow

/

mozilla/4.0+compatible+zyborg/1.0+

Rule

Path

Disallow

/

mozilla/5

Rule

Path

Disallow

/

mozilla/5.0+(compatible;+becomebot/3.0;++http://www.become.com/site_owners.html)

Rule

Path

Disallow

/

msiecrawler

Rule

Path

Disallow

/

msnbot-media

Rule

Path

Disallow

/

msr-isrccrawler

Rule

Path

Disallow

/

nambu

Rule

Path

Disallow

/

nationaldirectory-superspider

Rule

Path

Disallow

/

naverbot

Rule

Path

Disallow

/

naverbot

Rule

Path

Disallow

/

netants

Rule

Path

Disallow

/

netmechanic

Rule

Path

Disallow

/

nexen

Rule

Path

Disallow

/

nextgensearchbot

Rule

Path

Disallow

/

ng/2.0

Rule

Path

Disallow

/

nicebot

Rule

Path

Disallow

/

nicerspro

Rule

Path

Disallow

/

noxtrumbot/1.0

Rule

Path

Disallow

/

nutch

Rule

Path

Disallow

/

obot

Rule

Path

Disallow

/

ocelli

Rule

Path

Disallow

/

offline explorer

Rule

Path

Disallow

/

offline explorer/1.5

Rule

Path

Disallow

/

offline explorer/1.9

Rule

Path

Disallow

/

offline explorer/2.5

Rule

Path

Disallow

/

omniexplorer

Rule

Path

Disallow

/

omniexplorer_bot

Rule

Path

Disallow

/

omniexplorer_bot/1.07

Rule

Path

Disallow

/

omniexplorer_bot/1.09

Rule

Path

Disallow

/

omniexplorer_bot/1.10

Rule

Path

Disallow

/

ontownsbot

Rule

Path

Disallow

/

oozbot

Rule

Path

Disallow

/

openbot

Rule

Path

Disallow

/

openfind

Rule

Path

Disallow

/

openfind data gathere

Rule

Path

Disallow

/

openx

Rule

Path

Disallow

/

oracle ultra search

Rule

Path

Disallow

/

page_prefetcher

Rule

Path

Disallow

/

perl

Rule

Path

Disallow

/

perman

Rule

Path

Disallow

/

phpversion

Rule

Path

Disallow

/

piffany_web_cacher_v0.91

Rule

Path

Disallow

/

propowerbot/2.14

Rule

Path

Disallow

/

prowebwalker

Rule

Path

Disallow

/

psbot

Rule

Path

Disallow

/

psbot/0.1

Rule

Path

Disallow

/

psbot/0.1+(+http://www.picsearch.com/bot.html)

Rule

Path

Disallow

/

psibots

Rule

Path

Disallow

/

python-urllib

Rule

Path

Disallow

/

qeavis

Rule

Path

Disallow

/

queryn metasearch

Rule

Path

Disallow

/

radiation retriever 1.1

Rule

Path

Disallow

/

rankivabot

Rule

Path

Disallow

/

reap-crawler

Rule

Path

Disallow

/

repomonkey

Rule

Path

Disallow

/

repomonkey bait & tackle/v1.01

Rule

Path

Disallow

/

ricecomputerarchitecture

Rule

Path

Disallow

/

rma

Rule

Path

Disallow

/

robotgenius

Rule

Path

Disallow

/

robozilla

Rule

Path

Disallow

/

ru_com_viewer

Rule

Path

Disallow

/

rufusbot

Rule

Path

Disallow

/

sapphirewebcrawler

Rule

Path

Disallow

/

sbider

Rule

Path

Disallow

/

sbider

Rule

Path

Disallow

/

sbider/sbider-0.8.2-dev

Rule

Path

Disallow

/

scooter

Rule

Path

Disallow

/

scooter/2.0

Rule

Path

Disallow

/

scoutjet

Rule

Path

Disallow

/

scrubby

Rule

Path

Disallow

/

searchme

Rule

Path

Disallow

/

searchpreview

Rule

Path

Disallow

/

sensis web crawler

Rule

Path

Disallow

/

sensis web crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au)

Rule

Path

Disallow

/

sensis+web+crawler+

Rule

Path

Disallow

/

servage

Rule

Path

Disallow

/

servagerobot

Rule

Path

Disallow

/

setooz

Rule

Path

Disallow

/

shopwiki

Rule

Path

Disallow

/

sidewinder

Rule

Path

Disallow

/

sika

Rule

Path

Disallow

/

sitecat

Rule

Path

Disallow

/

sitecheck.internetseer.com

Rule

Path

Disallow

/

sitesnagger

Rule

Path

Disallow

/

slicehost

Rule

Path

Disallow

/

slurp.so/1.0

Rule

Path

Disallow

/

slurp@inktomi

Rule

Path

Disallow

/

slurp@inktomi.com

Rule

Path

Disallow

/

slurp/2.0-kitehourly

Rule

Path

Disallow

/

slurp/2.0-owlweekly

Rule

Path

Disallow

/

slurp/2.0j

Rule

Path

Disallow

/

slurp/3.0-au

Rule

Path

Disallow

/

slysearch

Rule

Path

Disallow

/

slysearch/1.0

Rule

Path

Disallow

/

slysearch/1.x

Rule

Path

Disallow

/

snapbot

Rule

Path

Disallow

/

snapbot/1.0

Rule

Path

Disallow

/

snappreviewbot

Rule

Path

Disallow

/

snappy

Rule

Path

Disallow

/

snappybot

Rule

Path

Disallow

/

sogou spider

Rule

Path

Disallow

/

sogou+spider

Rule

Path

Disallow

/

spankbot

Rule

Path

Disallow

/

spanner

Rule

Path

Disallow

/

speedy

Rule

Path

Disallow

/

speedyspider

Rule

Path

Disallow

/

sphsearch

Rule

Path

Disallow

/

sproose

Rule

Path

Disallow

/

stackrambler/2.0

Rule

Path

Disallow

/

stackrambler/2.0+(msie+incompatible)

Rule

Path

Disallow

/

superbot/4.4.0.60 (windows xp)

Rule

Path

Disallow

/

superpages

Rule

Path

Disallow

/

surveybot

Rule

Path

Disallow

/

surveybot/2.3

Rule

Path

Disallow

/

surveybot/2.3 (whois source)

Rule

Path

Disallow

/

suzuran

Rule

Path

Disallow

/

szukacz/1.4

Rule

Path

Disallow

/

t-online browser

Rule

Path

Disallow

/

t-rex

Rule

Path

Disallow

/

teleport

Rule

Path

Disallow

/

teleportpro

Rule

Path

Disallow

/

teleportpromiixpc

Rule

Path

Disallow

/

telesoft

Rule

Path

Disallow

/

teoma

Rule

Path

Disallow

/

the intraformant

Rule

Path

Disallow

/

thenomad

Rule

Path

Disallow

/

therarestparser

Rule

Path

Disallow

/

tighttwatbot

Rule

Path

Disallow

/

titan

Rule

Path

Disallow

/

tocrawl/urldispatcher

Rule

Path

Disallow

/

trident

Rule

Path

Disallow

/

true_robot

Rule

Path

Disallow

/

true_robot /1.0turingos

Rule

Path

Disallow

/

true_robot/1.0

Rule

Path

Disallow

/

turingos

Rule

Path

Disallow

/

turnitinbot

Rule

Path

Disallow

/

turnitinbot

Rule

Path

Disallow

/

turnitinbot/1.5

Rule

Path

Disallow

/

turnitinbot/2.0 http://www.turnitin.com/robot/crawlerinfo.html

Rule

Path

Disallow

/

tutorial crawler

Rule

Path

Disallow

/

twiceler

Rule

Path

Disallow

/

twiceler

Rule

Path

Disallow

/

twiceler-0.9

Rule

Path

Disallow

/

twiceler-0.9+http://www.cuill.com/twiceler/robot.html

Rule

Path

Disallow

/

uberbot

Rule

Path

Disallow

/

ultraseek

Rule

Path

Disallow

/

url control

Rule

Path

Disallow

/

url_spider_pro

Rule

Path

Disallow

/

urly warning

Rule

Path

Disallow

/

user-agetn: mozilla/4.0 (compatible; b-l-i-t-z-b-o-t)

Rule

Path

Disallow

/

uw_cse_xwc

Rule

Path

Disallow

/

vci

Rule

Path

Disallow

/

vci webviewer vci webviewer win32

Rule

Path

Disallow

/

vcivci

Rule

Path

Disallow

/

violabot

Rule

Path

Disallow

/

voilabot

Rule

Path

Disallow

/

voyager

Rule

Path

Disallow

/

voyager/1.0

Rule

Path

Disallow

/

voyager/2.0

Rule

Path

Disallow

/

vscooter

Rule

Path

Disallow

/

vwbot

Rule

Path

Disallow

/

web downloader/4.5

Rule

Path

Disallow

/

web downloader/5.7

Rule

Path

Disallow

/

web downloader/6.9

Rule

Path

Disallow

/

web image collector

Rule

Path

Disallow

/

webalta

Rule

Path

Disallow

/

webaroobot

Rule

Path

Disallow

/

webauto

Rule

Path

Disallow

/

webbandit

Rule

Path

Disallow

/

webbandit/3.50

Rule

Path

Disallow

/

webcollage

Rule

Path

Disallow

/

webcopier

Rule

Path

Disallow

/

webcopier v3.2a

Rule

Path

Disallow

/

webcopier v3.3

Rule

Path

Disallow

/

webcopier v3.3.0

Rule

Path

Disallow

/

webcopier v3.3.2

Rule

Path

Disallow

/

webcrawler

Rule

Path

Disallow

/

webdatacentrebot

Rule

Path

Disallow

/

webenhancer

Rule

Path

Disallow

/

webmasterworld extractor

Rule

Path

Disallow

/

webmasterworldforumbot

Rule

Path

Disallow

/

websauger

Rule

Path

Disallow

/

websense

Rule

Path

Disallow

/

website quester

Rule

Path

Disallow

/

webspider

Rule

Path

Disallow

/

webster pro

Rule

Path

Disallow

/

webstripper

Rule

Path

Disallow

/

webviewer

Rule

Path

Disallow

/

webzip

Rule

Path

Disallow

/

webzip/3.5 (http://www.spidersoft.com)

Rule

Path

Disallow

/

webzip/4.0

Rule

Path

Disallow

/

wget

Rule

Path

Disallow

/

wget

Rule

Path

Disallow

/

wget/1.10.2

Rule

Path

Disallow

/

wget/1.5.3

Rule

Path

Disallow

/

wget/1.6

Rule

Path

Disallow

/

wget/1.9.1

Rule

Path

Disallow

/

wget/1.9+cvs-stable (red hat modified)

Rule

Path

Disallow

/

whois.de

Rule

Path

Disallow

/

whoisde.de

Rule

Path

Disallow

/

willybot

Rule

Path

Disallow

/

win32

Rule

Path

Disallow

/

worio

Rule

Path

Disallow

/

www-collector-e

Rule

Path

Disallow

/

xenu link sleuth

Rule

Path

Disallow

/

xenu link sleuth 1.2i

Rule

Path

Disallow

/

xenu's

Rule

Path

Disallow

/

xenu's link sleuth 1.1c

Rule

Path

Disallow

/

y!j

Rule

Path

Disallow

/

y!j-bsc

Rule

Path

Disallow

/

y!j-bsc

Rule

Path

Disallow

/

y!j-bsc/1.0

Rule

Path

Disallow

/

y!j/1.0

Rule

Path

Disallow

/

yanga

Rule

Path

Disallow

/

yeti

Rule

Path

Disallow

/

yooglifetchagent/0.1

Rule

Path

Disallow

/

zao

Rule

Path

Disallow

/

zermelo

Rule

Path

Disallow

/

zeus

Rule

Path

Disallow

/

zeus 32297 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus link scout

Rule

Path

Disallow

/

zibber

Rule

Path

Disallow

/

zibber v0.1

Rule

Path

Disallow

/

zibber-v0.1(www.zibb.com/crawler/)

Rule

Path

Disallow

/

zupeecrawler

Rule

Path

Disallow

/

zyborg

Rule

Path

Disallow

/

zyborg/1.0

Rule

Path

Disallow

/

Site Domain	helaldunyamarketleri.com
Base Domain	helaldunyamarketleri.com
Scan Status	Ok
Last Scan	2024-09-15T20:06:26+00:00
Next Scan	2024-10-15T20:06:26+00:00

Scanned	2024-09-15T20:06:26+00:00
URL	https://helaldunyamarketleri.com/robots.txt
Domain IPs	94.199.202.166
Response IP	94.199.202.166
Found	Yes
Hash	21e88d7c58d7f242c39d18dc992b22d1ca0cf179c5ebac9205a6338f36e05e07
SimHash	30975053daa3

helaldunyamarketleri.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

googlebot

googlebot-image

ahrefsbot

sp_auditbot

riddler

blp_bbot/0.1

user-agent

compatible ;

mozilla

libwww

lwp-trivial

curl

php/

urllib

gt::www

snoopy

mfc_tear_sample

http::lite

phpcrawl

uri::fetch

zend_http_client

http client

pecl::http

panscient.com

ibm evv

bork-edition

fetch api request

wep search

wells search ii

missigua locator

indy library

java/1.4.1_04

mj12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)

java/1.6.0_04

java/1.5.0_06

java/1.6.0_11

java/1.6.0_07

java/1.6.0_13

isc systems irc search 2.1

mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)

aisearchbot

alexibot

amazoncloudservices

apexoo spider 1.0

aqua_products

arachnoidea

architextspider

aspsimply

asterias

attributorcorporationdmcabot

avant+browser+

axonize-bot

b2w/0.1

backdoorbot/1.0

baiduspider

baiduspider+(+http://www.baidu.com/search/spider_jp.html)

becomebot

becomebot/3.0

bitacle

bitacle bot

bitacle bot/1.1

bitacle*

black hole

blowfish/1.0

bobcrawl

boitho

boitho.com

bookmark search tool

bot/1.0

bot/1.0+(bot;+http://;+bot@bot.bot)

botalot

botmobi

bsalsa

builtbottough

bullseye/1.0

helaldunyamarketleri.com
robots.txt