thianthong.com robots.txt

*

Rule	Path
Disallow	/BingSiteAuth.xml
Disallow	/cgi-bin/
Disallow	/ekonomirubriker_wp.html
Disallow	/?
Disallow	/app/
Disallow	/bin/
Disallow	/dev/
Disallow	/lib/
Disallow	/phpserver/
Disallow	/pkginfo/
Disallow	/report/
Disallow	/setup/
Disallow	/update/
Disallow	/var/
Disallow	/vendor/
Disallow	/catalog/product_compare/
Disallow	/searchautocomplete/
Disallow	/checkout/
Disallow	/control/
Disallow	/contacts/
Disallow	/customer/
Disallow	/customize/
Disallow	/newsletter/
Disallow	/review/
Disallow	/sendfriend/
Disallow	/wishlist/
Disallow	/composer.json
Disallow	/composer.lock
Disallow	/CONTRIBUTING.md
Disallow	/CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow	/COPYING.txt
Disallow	/Gruntfile.js
Disallow	/LICENSE.txt
Disallow	/LICENSE_AFL.txt
Disallow	/nginx.conf.sample
Disallow	/package.json
Disallow	/php.ini.sample
Disallow	/RELEASE_NOTES.txt
Disallow	/?product_list_mode=
Disallow	/?product_list_order=
Disallow	/?product_list_limit=
Disallow	/?product_list_dir=
Disallow	/*?SID=
Disallow	/*?
Disallow	/*.php$
Disallow	/*.CVS
Disallow	/*.Zip$
Disallow	/*.Svn$
Disallow	/*.Idea$
Disallow	/*.Sql$
Disallow	/*.Tgz$

Rule

Path

Disallow

/BingSiteAuth.xml

Disallow

/cgi-bin/

Disallow

/ekonomirubriker_wp.html

Disallow

/*?*

Disallow

/app/

Disallow

/bin/

Disallow

/dev/

Disallow

/lib/

Disallow

/phpserver/

Disallow

/pkginfo/

Disallow

/report/

Disallow

/setup/

Disallow

/update/

Disallow

/var/

Disallow

/vendor/

Disallow

/catalog/product_compare/

Disallow

/searchautocomplete/

Disallow

/checkout/

Disallow

/control/

Disallow

/contacts/

Disallow

/customer/

Disallow

/customize/

Disallow

/newsletter/

Disallow

/review/

Disallow

/sendfriend/

Disallow

/wishlist/

Disallow

/composer.json

Disallow

/composer.lock

Disallow

/CONTRIBUTING.md

Disallow

/CONTRIBUTOR_LICENSE_AGREEMENT.html

Disallow

/COPYING.txt

Disallow

/Gruntfile.js

Disallow

/LICENSE.txt

Disallow

/LICENSE_AFL.txt

Disallow

/nginx.conf.sample

Disallow

/package.json

Disallow

/php.ini.sample

Disallow

/RELEASE_NOTES.txt

Disallow

/*?*product_list_mode=

Disallow

/*?*product_list_order=

Disallow

/*?*product_list_limit=

Disallow

/*?*product_list_dir=

Disallow

/*?SID=

Disallow

/*?

Disallow

/*.php$

Disallow

/*.CVS

Disallow

/*.Zip$

Disallow

/*.Svn$

Disallow

/*.Idea$

Disallow

/*.Sql$

Disallow

/*.Tgz$

abonti

Rule	Path
Disallow	/

Rule

Path

Disallow

/

abonti/0.92

Rule	Path
Disallow	/

Rule

Path

Disallow

/

abot v1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aboutthedomain

Rule	Path
Disallow	/

Rule

Path

Disallow

/

add catalog

Rule	Path
Disallow	/

Rule

Path

Disallow

/

add catalog/2.1

Rule	Path
Disallow	/

Rule

Path

Disallow

/

advbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

advbot/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ahrefs-bot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ahrefsbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ahrefs-bot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ahrefs-bot/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ahrefs-bot/3.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ahrefs-bot/4.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ahrefs-bot/5.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aihitbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aihitbot/2.9

Rule	Path
Disallow	/

Rule

Path

Disallow

/

anonymous/0.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

arachnida

Rule	Path
Disallow	/

Rule

Path

Disallow

/

associative spider

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baidu spider

Rule	Path
Disallow	/

Rule

Path

Disallow

/

battleztar bazinga

Rule	Path
Disallow	/

Rule

Path

Disallow

/

battleztar bazinga/0.01

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bdfetch

Rule	Path
Disallow	/

Rule

Path

Disallow

/

betabot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bieshu

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bigli seo

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blackboard safeassign

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blazer 1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blexbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blp_bbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blp_bbot/0.1

Rule	Path
Disallow	/

Rule

Path

Disallow

/

boia-accessibility-agent/pr 1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bot for jce

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bot/0.1 (bot for jce)

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bpimagewalker

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bpimagewalker/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bubing

Rule	Path
Disallow	/

Rule

Path

Disallow

/

buibui-bot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

buibui-bot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ca-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ca-crawler/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

cakephp

Rule	Path
Disallow	/

Rule

Path

Disallow

/

calypso v/0.01

Rule	Path
Disallow	/

Rule

Path

Disallow

/

calypso

Rule

Path

Disallow

/

cb/nutch-1.7

Rule

Path

Disallow

/

ccbot

Rule

Path

Disallow

/

ccbot/2.0

Rule

Path

Disallow

/

checkbot

Rule

Path

Disallow

/

checkgzipcompression.com

Rule

Path

Disallow

/

chushou

Rule

Path

Disallow

/

cloudservermarketspider

Rule

Path

Disallow

/

cloudservermarketspider/1.0

Rule

Path

Disallow

/

clushbot/3.x-binaryfury

Rule

Path

Disallow

/

cms crawler

Rule

Path

Disallow

/

cms crawler: http://www.cmscrawler.com

Rule

Path

Disallow

/

coccoc

Rule

Path

Disallow

/

coincornerbot

Rule

Path

Disallow

/

coincornerbot/1.1

Rule

Path

Disallow

/

copyscape

Rule

Path

Disallow

/

crawler4j

Rule

Path

Disallow

/

crazywebcrawler 0.9.0

Rule

Path

Disallow

/

crazywebcrawler 0.9.1

Rule

Path

Disallow

/

crazywebcrawler 0.9.7

Rule

Path

Disallow

/

crazywebcrawler

Rule

Path

Disallow

/

crazywebcrawler-spider

Rule

Path

Disallow

/

crowsnest

Rule

Path

Disallow

/

crowsnest/0.5

Rule

Path

Disallow

/

curious george - www.analyticsseo.com/crawler

Rule

Path

Disallow

/

curious george

Rule

Path

Disallow

/

cuwhois

Rule

Path

Disallow

/

cuwhois/1.0

Rule

Path

Disallow

/

dahoms

Rule

Path

Disallow

/

datagnionbot

Rule

Path

Disallow

/

deusu/5.0.2

Rule

Path

Disallow

/

digincore

Rule

Path

Disallow

/

digincore bot

Rule

Path

Disallow

/

dispatch/0.11.0

Rule

Path

Disallow

/

domain re-animator bot

Rule

Path

Disallow

/

domainappender /1.0

Rule

Path

Disallow

/

domainappender

Rule

Path

Disallow

/

domaincrawler/3.0

Rule

Path

Disallow

/

domainsigmacrawler

Rule

Path

Disallow

/

domainsigmacrawler/0.1

Rule

Path

Disallow

/

domnutch

Rule

Path

Disallow

/

domnutch-bot

Rule

Path

Disallow

/

domnutch-bot/nutch

Rule

Path

Disallow

/

domnutch-bot/nutch-1.0

Rule

Path

Disallow

/

dotbot

Rule

Path

Disallow

/

eccp/1.2.1

Rule

Path

Disallow

/

ecommercebot

Rule

Path

Disallow

/

enlle punto com/nutch-1.9

Rule

Path

Disallow

/

episerver link checker

Rule

Path

Disallow

/

euripbot

Rule

Path

Disallow

/

euripbot/2.0

Rule

Path

Disallow

/

evc/2.0

Rule

Path

Disallow

/

evc-batch

Rule

Path

Disallow

/

evc-batch/2.0

Rule

Path

Disallow

/

express webpictures

Rule

Path

Disallow

/

faraday v0.8.8

Rule

Path

Disallow

/

faraday

Rule

Path

Disallow

/

findxbot

Rule

Path

Disallow

/

findxbot/1.0

Rule

Path

Disallow

/

flamingo_searchengine

Rule

Path

Disallow

/

flipboard robot

Rule

Path

Disallow

/

getproxi.es-bot

Rule

Path

Disallow

/

getproxi.es-bot/1.1

Rule

Path

Disallow

/

gigablastopensource

Rule

Path

Disallow

/

gigablastopensource/1.0

Rule

Path

Disallow

/

girafabot

Rule

Path

Disallow

/

gluten free crawler

Rule

Path

Disallow

/

gluten free crawler/1.0

Rule

Path

Disallow

/

griffinbot

Rule

Path

Disallow

/

grifinbot/0.01

Rule

Path

Disallow

/

gwpimages

Rule

Path

Disallow

/

gwpimages/1.0

Rule

Path

Disallow

/

haiula

Rule

Path

Disallow

/

haiula/1.4

Rule

Path

Disallow

/

haosouspider

Rule

Path

Disallow

/

hivemind

Rule

Path

Disallow

/

hostharvest

Rule

Path

Disallow

/

hostharvest/0.4.28

Rule

Path

Disallow

/

hrcrawler

Rule

Path

Disallow

/

hrcrawler/2.0

Rule

Path

Disallow

/

http://git.io/tl_s2w

Rule

Path

Disallow

/

http://www.checkprivacy.or.kr:6600/rs/privacy_enfaq.jsp

Rule

Path

Disallow

/

hubspot links crawler 1.0

Rule

Path

Disallow

/

hubspot webcrawler

Rule

Path

Disallow

/

hubspot

Rule

Path

Disallow

/

hunchan

Rule

Path

Disallow

/

hypercrawl

Rule

Path

Disallow

/

hypercrawl/0.2

Rule

Path

Disallow

/

icap-iod

Rule

Path

Disallow

/

icc-crawler

Rule

Path

Disallow

/

icc-crawler/2.0

Rule

Path

Disallow

/

ichiro robot

Rule

Path

Disallow

/

image.coccoc/1.0

Rule

Path

Disallow

/

image2play

Rule

Path

Disallow

/

image2play/0.1

Rule

Path

Disallow

/

indy library

Rule

Path

Disallow

/

insightscollector

Rule

Path

Disallow

/

insightscollector/0.1

Rule

Path

Disallow

/

insightscollector/0.1beta

Rule

Path

Disallow

/

integrity/5

Rule

Path

Disallow

/

internaetboten

Rule

Path

Disallow

/

internaetboten/0.99

Rule

Path

Disallow

/

irl crawler

Rule

Path

Disallow

/

james bot - webcrawler

Rule

Path

Disallow

/

james bot

Rule

Path

Disallow

/

jamesbot

Rule

Path

Disallow

/

jetbrains 5.0

Rule

Path

Disallow

/

jetbrains

Rule

Path

Disallow

/

kraken

Rule

Path

Disallow

/

kraken/0.1

Rule

Path

Disallow

/

kyoto-tohoku-crawler/v1

Rule

Path

Disallow

/

larbin

Rule

Path

Disallow

/

lechenie

Rule

Path

Disallow

/

libwww-perl

Rule

Path

Disallow

/

link checker

Rule

Path

Disallow

/

link/1.0

Rule

Path

Disallow

/

linkcheck

Rule

Path

Disallow

/

linkcheckv3.0

Rule

Path

Disallow

/

linkdex

Rule

Path

Disallow

/

linkdex.com/v2.0

Rule

Path

Disallow

/

linkdex.com/v2.1

Rule

Path

Disallow

/

linkdexbot

Rule

Path

Disallow

/

linkdexbot/2.0

Rule

Path

Disallow

/

linkdexbot/2.1

Rule

Path

Disallow

/

linkdexbot-mobile/2.1

Rule

Path

Disallow

/

linkpadbot

Rule

Path

Disallow

/

linkpadbot/1.06

Rule

Path

Disallow

/

linqiascrapebot

Rule

Path

Disallow

/

linqiascrapebot/1.0

Rule

Path

Disallow

/

lipperhey seo service

Rule

Path

Disallow

/

lipperhey

Rule

Path

Disallow

/

lipperhey-kaus-australis

Rule

Path

Disallow

/

lipperhey-kaus-australis/5.0

Rule

Path

Disallow

/

listicka

Rule

Path

Disallow

/

lssrocketcrawler

Rule

Path

Disallow

/

lssrocketcrawler/1.0 lightspeedsystems

Rule

Path

Disallow

/

lssrocketcrawler/1.0

Rule

Path

Disallow

/

ltx71

Rule

Path

Disallow

/

lwnutch/nutch-1.4

Rule

Path

Disallow

/

mail.ru

Rule

Path

Disallow

/

mail.ru_bot

Rule

Path

Disallow

/

mail.ru_bot/2.0

Rule

Path

Disallow

/

mail.ru_bot/fast/2.0

Rule

Path

Disallow

/

md5sum

Rule

Path

Disallow

/

md5sum\x22

Rule

Path

Disallow

/

meanpathbot

Rule

Path

Disallow

/

megaindex.ru

Rule

Path

Disallow

/

megaindex.ru/2.0

Rule

Path

Disallow

/

mezhpozvonochnoi

Rule

Path

Disallow

/

mike-crawler

Rule

Path

Disallow

/

mixbot

Rule

Path

Disallow

/

mixrankbot

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

monkeybot/0.1

Rule

Path

Disallow

/

my crawler

Rule

Path

Disallow

/

my nutch spider/nutch-1.9

Rule

Path

Disallow

/

mycrowl/nutch-1.9

Rule

Path

Disallow

/

mygreatua/2.0

Rule

Path

Disallow

/

myiptest

Rule

Path

Disallow

/

nameprotect robot

Rule

Path

Disallow

/

nerdybot

Rule

Path

Disallow

/

netcraft spider

Rule

Path

Disallow

/

netestate ne crawler

Rule

Path

Disallow

/

netlyzer fastprobe

Rule

Path

Disallow

/

netresearchserver

Rule

Path

Disallow

/

netresearchserver/4.0

Rule

Path

Disallow

/

nmap scripting engine

Rule

Path

Disallow

/

node.io

Rule

Path

Disallow

/

node.js

Rule

Path

Disallow

/

node/simplecrawler 0.5.2

Rule

Path

Disallow

/

node/simplecrawler

Rule

Path

Disallow

/

obot/2.3.1

Rule

Path

Disallow

/

omgilibot

Rule

Path

Disallow

/

omgilibot/0.4

Rule

Path

Disallow

/

online domain tools - online website link checker

Rule

Path

Disallow

/

online domain tools - online website link checker/1.2

Rule

Path

Disallow

/

openfind robot

Rule

Path

Disallow

/

openhosebot

Rule

Path

Disallow

/

openhosebot/2.1

Rule

Path

Disallow

/

openstat

Rule

Path

Disallow

/

openstat/0.1

Rule

Path

Disallow

/

optimizationcrawler

Rule

Path

Disallow

/

optimizationcrawler/0.2

Rule

Path

Disallow

/

page analyzer v4.0

Rule

Path

Disallow

/

page analyzer

Rule

Path

Disallow

/

pageanalyzer

Rule

Path

Disallow

/

pageanalyzer/1.1

Rule

Path

Disallow

/

pageanalyzer/1.5

Rule

Path

Disallow

/

pagesinventory

Rule

Path

Disallow

/

pagespeed/1.1 fetcher

Rule

Path

Disallow

/

pagespeed/1.1

Rule

Path

Disallow

/

pagespeedbot

Rule

Path

Disallow

/

perl lwp

Rule

Path

Disallow

/

phpcrawl

Rule

Path

Disallow

/

phpsitecheck 1.0

Rule

Path

Disallow

/

phpsitecheck

Rule

Path

Disallow

/

plukkie

Rule

Path

Disallow

/

pogs/2.0

Rule

Path

Disallow

/

powermarks

Rule

Path

Disallow

/

powerpivot

Rule

Path

Disallow

/

privacy_enfaq.jsp

Rule

Path

Disallow

/

prlog

Rule

Path

Disallow

/

prlog/1.0

Rule

Path

Disallow

/

publiclibraryarchive.org

Rule

Path

Disallow

/

publiclibraryarchive.org/1.0

Rule

Path

Disallow

/

pu_in crawler

Rule

Path

Disallow

/

putin

Rule

Path

Disallow

/

putin spider

Rule

Path

Disallow

/

qingdao

Rule

Path

Disallow

/

qlikview

Rule

Path

Disallow

/

quipu

Rule

Path

Disallow

/

quipu/1.0

Rule

Path

Disallow

/

quipu/2.0

Rule

Path

Disallow

/

r6_commentreader

Rule

Path

Disallow

/

r6_feedfetcher

Rule

Path

Disallow

/

riddler

Rule

Path

Disallow

/

rivalseek.com-bot

Rule

Path

Disallow

/

rogerbot

Rule

Path

Disallow

/

rogerbot/1.0

Rule

Path

Disallow

/

rootlink

Rule

Path

Disallow

/

ru_bot/2.0

Rule

Path

Disallow

/

scopia

Rule

Path

Disallow

/

scopia crawler

Rule

Path

Disallow

/

scopia crawler 1.0

Rule

Path

Disallow

/

scopia crawler 1.1

Rule

Path

Disallow

/

scopia crawler 1.2

Rule

Path

Disallow

/

scrapy

Rule

Path

Disallow

/

scrapy/0.16.5

Rule

Path

Disallow

/

scrapy/0.24.4

Rule

Path

Disallow

/

scrapy/0.24.5

Rule

Path

Disallow

/

scrapy/0.24.6

Rule

Path

Disallow

/

scrapy/1.0.1

Rule

Path

Disallow

/

screaming frog seo spider

Rule

Path

Disallow

/

screaming frog seo spider/2,55

Rule

Path

Disallow

/

screaming frog seo spider/2.55

Rule

Path

Disallow

/

screaming frog seo spider/3.1

Rule

Path

Disallow

/

screaming frog seo spider/3.3

Rule

Path

Disallow

/

screaming frog seo spider/4.1

Rule

Path

Disallow

/

screaming frog seo spider/5.0

Rule

Path

Disallow

/

screaming frog seo spider/5.1

Rule

Path

Disallow

/

screaming frog seo spider/5.1 beta 2

Rule

Path

Disallow

/

scrutiny/4

Rule

Path

Disallow

/

semrushbot

Rule

Path

Disallow

/

semrushbot-sa

Rule

Path

Disallow

/

seodiver/1.0

Rule

Path

Disallow

/

seokicks

Rule

Path

Disallow

/

seokicks-robot

Rule

Path

Disallow

/

seolyticscrawler

Rule

Path

Disallow

/

seolyticscrawler/3.0

Rule

Path

Disallow

/

seoscanners

Rule

Path

Disallow

/

seoscanners.net/1

Rule

Path

Disallow

/

seostats 2.1.0

Rule

Path

Disallow

/

seosys/nutch-2.3

Rule

Path

Disallow

/

setcronjob/1.0

Rule

Path

Disallow

/

seznambot

Rule

Path

Disallow

/

sheerboredom.experimental.robot

Rule

Path

Disallow

/

sheerboredom.experimental.robot/0.2

Rule

Path

Disallow

/

showyoubot

Rule

Path

Disallow

/

simplecrawler

Rule

Path

Disallow

/

sistrix crawler

Rule

Path

Disallow

/

sistrix

Rule

Path

Disallow

/

sitebot

Rule

Path

Disallow

/

sitebot/0.1

Rule

Path

Disallow

/

siteexplorer

Rule

Path

Disallow

/

siteexplorer/1.0

Rule

Path

Disallow

/

siteexplorer/1.0b

Rule

Path

Disallow

/

siteluxbot

Rule

Path

Disallow

/

siteluxbot/1.0

Rule

Path

Disallow

/

skimbot

Rule

Path

Disallow

/

skimbot/1.0

Rule

Path

Disallow

/

sky nutch crawler/nutch-1.9

Rule

Path

Disallow

/

smtbot

Rule

Path

Disallow

/

smtbot/1.0

Rule

Path

Disallow

/

snk screenshot bot

Rule

Path

Disallow

/

snk screenshot bot/0.20

Rule

Path

Disallow

/

sogou spider

Rule

Path

Disallow

/

sogou web spider

Rule

Path

Disallow

/

spambayes

Rule

Path

Disallow

/

spambayes/1.1a3+

Rule

Path

Disallow

/

spbot

Rule

Path

Disallow

/

spbot/4.4.2

Rule

Path

Disallow

/

spiderbot

Rule

Path

Disallow

/

spiderling

Rule

Path

Disallow

/

spiderbot/nutch-1.7

Rule

Path

Disallow

/

spray-can

Rule

Path

Disallow

/

spray-can/1.2.1

Rule

Path

Disallow

/

ssg/3.0

Rule

Path

Disallow

/

statastico

Rule

Path

Disallow

/

statastico/4.0

Rule

Path

Disallow

/

steeler

Rule

Path

Disallow

/

steeler/3.5

Rule

Path

Disallow

/

stratagems kumo

Rule

Path

Disallow

/

stratagems

Rule

Path

Disallow

/

studiofaca search

Rule

Path

Disallow

/

studiofaca

Rule

Path

Disallow

/

sukibot

Rule

Path

Disallow

/

sukibot_heritrix

Rule

Path

Disallow

/

sukibot_heritrix/3.1.1

Rule

Path

Disallow

/

surveybot

Rule

Path

Disallow

/

synapse

Rule

Path

Disallow

/

synthesio crawler release monalisa

Rule

Path

Disallow

/

tbot-nutch/nutch-1.10

Rule

Path

Disallow

/

traackr.com bot

Rule

Path

Disallow

/

trendictionbot

Rule

Path

Disallow

/

trendiction-bot

Rule

Path

Disallow

/

truebot

Rule

Path

Disallow

/

truebot/1.0

Rule

Path

Disallow

/

tulipchain/5.xx

Rule

Path

Disallow

/

twmbot/0.1

Rule

Path

Disallow

/

typhoeus

Rule

Path

Disallow

/

ucmore crawler app

Rule

Path

Disallow

/

umbot-ln

Rule

Path

Disallow

/

umbot-ln/1.0

Rule

Path

Disallow

/

updown_tester

Rule

Path

Disallow

/

urlchecker

Rule

Path

Disallow

/

v1.0/1.2

Rule

Path

Disallow

/

w3af.org

Rule

Path

Disallow

/

wasalive

Rule

Path

Disallow

/

wasalive-bot

Rule

Path

Disallow

/

vbseo

Rule

Path

Disallow

/

wbsearchbot

Rule

Path

Disallow

/

wbsearchbot/1.1

Rule

Path

Disallow

/

wearenotevil

Rule

Path

Disallow

/

webalta

Rule

Path

Disallow

/

webalta crawler

Rule

Path

Disallow

/

web corpus crawler

Rule

Path

Disallow

/

webcookies

Rule

Path

Disallow

/

webcookies/1.0

Rule

Path

Disallow

/

webcopier vx.xa

Rule

Path

Disallow

/

webnest 0.9

Rule

Path

Disallow

/

webql

Rule

Path

Disallow

/

webscout

Rule

Path

Disallow

/

webscout/1.0

Rule

Path

Disallow

/

web-sniffer

Rule

Path

Disallow

/

web-sniffer/1.1.0

Rule

Path

Disallow

/

webster pro v3.4

Rule

Path

Disallow

/

webtarantula.com crawler

Rule

Path

Disallow

/

wecrawlforthepeace

Rule

Path

Disallow

/

vegebot

Rule

Path

Disallow

/

vegi bot

Rule

Path

Disallow

/

welikelinks

Rule

Path

Disallow

/

vericitecrawler

Rule

Path

Disallow

/

vericitecrawler/nutch-1.9

Rule

Path

Disallow

/

whatweb

Rule

Path

Disallow

/

whatweb/0.4.8-dev

Rule

Path

Disallow

/

visited by http://tools.geek-tools.org

Rule

Path

Disallow

/

voila robot

Rule

Path

Disallow

/

voltron

Rule

Path

Disallow

/

woobot

Rule

Path

Disallow

/

woobot/1.1

Rule

Path

Disallow

/

woobot/2.0

Rule

Path

Disallow

/

vorboss web crawler

Rule

Path

Disallow

/

vorboss web crawler/nutch-2.3

Rule

Path

Disallow

/

worldbrewbot

Rule

Path

Disallow

/

worldbrewbot/2.1

Rule

Path

Disallow

/

worldwebheritage.org

Rule

Path

Disallow

/

worldwebheritage.org/1.0

Rule

Path

Disallow

/

wscheck.com

Rule

Path

Disallow

/

wscheck.com/1.0.0

Rule

Path

Disallow

/

www.deadlinkchecker.com

Rule

Path

Disallow

/

www.petitsage.fr site detector 0.4

Rule

Path

Disallow

/

www-mechanize

Rule

Path

Disallow

/

www-mechanize/1.74

Rule

Path

Disallow

/

xenu link sleuth

Rule

Path

Disallow

/

xenu's link sleuth

Rule

Path

Disallow

/

xovibot

Rule

Path

Disallow

/

xovibot/2.0

Rule

Path

Disallow

/

xspider

Rule

Path

Disallow

/

yandex robot

Rule

Path

Disallow

/

yandex

Rule

Path

Disallow

/

yetibot

Rule

Path

Disallow

/

yisouspider

Rule

Path

Disallow

/

yoozbot

Rule

Path

Disallow

/

yoozbot-2.2

Rule

Path

Disallow

/

zgrab/0.x

Rule

Path

Disallow

/

zzabmbot

Rule

Path

Disallow

/

zzabmbot/1.0

Rule

Path

Disallow

/

Site Domain	thianthong.com
Base Domain	thianthong.com
Scan Status	Ok
Last Scan	2024-10-23T02:05:21+00:00
Next Scan	2024-11-22T02:05:21+00:00

Scanned	2024-10-23T02:05:21+00:00
URL	https://thianthong.com/robots.txt
Domain IPs	104.21.13.61, 172.67.198.146, 2606:4700:3031::6815:d3d, 2606:4700:3032::ac43:c692
Response IP	104.21.13.61
Found	Yes
Hash	22670f94a634ddd82ea8f71a605909f0444f7854990ee7e2cc655bc7213bd8d9
SimHash	7ad14916e6b8

thianthong.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

abonti

abonti/0.92

abot v1.0

aboutthedomain

add catalog

add catalog/2.1

advbot

advbot/2.0

ahrefsbot

ahrefs-bot

ahrefsbot/1.0

ahrefs-bot/1.0

ahrefs-bot/2.0

ahrefs-bot/3.0

ahrefs-bot/4.0

ahrefs-bot/5.0

aihitbot

aihitbot/2.9

anonymous/0.0

arachnida

associative spider

baiduspider

baidu spider

battleztar bazinga

battleztar bazinga/0.01

bdfetch

betabot

bieshu

bigli seo

blackboard safeassign

blazer 1.0

blexbot

blexbot/1.0

blp_bbot

blp_bbot/0.1

boia-accessibility-agent/pr 1.0

bot for jce

bot/0.1 (bot for jce)

bpimagewalker

bpimagewalker/2.0

bubing

buibui-bot

buibui-bot/1.0

ca-crawler

ca-crawler/1.0

cakephp

calypso v/0.01

calypso

cb/nutch-1.7

ccbot

ccbot/2.0

checkbot

checkgzipcompression.com

chushou

cloudservermarketspider

cloudservermarketspider/1.0

clushbot/3.x-binaryfury

cms crawler

cms crawler: http://www.cmscrawler.com

coccoc

coincornerbot

coincornerbot/1.1

copyscape

crawler4j

crazywebcrawler 0.9.0

crazywebcrawler 0.9.1

crazywebcrawler 0.9.7

crazywebcrawler

crazywebcrawler-spider

crowsnest

crowsnest/0.5

curious george - www.analyticsseo.com/crawler

curious george

cuwhois

thianthong.com
robots.txt