detikindonesia.co.id
robots.txt

Groups

rogerbot
exabot
mj12bot
semrushbot
dotbot
gigabot
blackwidow
chinaclaw
custo
disco
download\ demon
ecatch
eirgrabber
emailsiphon
emailwolf
express\ webpictures
extractorpro
eyenetie
flashget
getright
getweb!
go!zilla
go-ahead-got-it
grabnet
grafula
hmview
httrack
image\ stripper
image\ sucker
indy\ library
interget
internet\ ninja
jetcar
joc\ web\ spider
larbin
leechftp
mass\ downloader
midown\ tool
mister\ pix
navroad
nearsite
netants
netspider
net\ vampire
netzip
octopus
offline\ explorer
offline\ navigator
pagegrabber
papa\ foto
pavuk
pcbrowser
realdownload
reget
sitesnagger
smartdownload
superbot
superhttp
surfbot
takeout
teleport\ pro
voideye
web\ image\ collector
web\ sucker
webauto
webcopier
webfetch
webgo\ is
webleacher
webreaper
websauger
website\ extractor
website\ quester
webstripper
webwhacker
webzip
wget
widow
wwwoffle
xaldon\ webspider
zeus
turnitinbot
accessible web bot
addthis
adyen
ahrefs site audit
ahrefsbot
alertsite by smartbear
alexa archive
alexa site audit
amazon contxbot
amazonbot
applebot
audisto crawler
automaton
awario
baidu
barkrowler
better uptime
ahrefssiteaudit
bing ads
bing preview
bingbot
blexbot
blockaid
blogvault
bluesky link preview service
botify
brandwatch
chargebee
clickagy
cludo
cốc cốc
contentking
cookiebot
coveo bot
criteobot
cxense
dataforseo
dataforseo bot
deadlinkchecker
detectify
dr. link check
drata autopilot
duckduckbot
easycron
elmah.io uptime monitoring
emoney advisor
erepublik.tools
evouptimebot
exodus
feedbin
feeder
feedly
feedwind crawler
freshping
fullstory
ghost inspector
gptbot
grapeshot
gtmetrix
hetrixtools
honeybadgerbot
hosttracker
ias crawler
iframely
innologica
integromat
internet archive
internetarchivebot
jagged pixel uptimebot
jetpack
kagi bot
klaviyo
let's encrypt
linkedinbot
mainwp
managewp
marginalia search
marketgoo
mars finder
medialogia bot
mediamonitoringbot
metorik
mgidbot
missinglettr bot
modular ds
mojeek
monitorss
moz dotbot
moz rogerbot
msn
new relic
newsblur
nodeping
notabot
oh dear
outbrain
ozon web grabber
parse.ly
paypal
petalbot
pingdom
pinterestbot
prerender
pressengine bot
proximic
qualys
quantcastbot
qwant
rackspace
readable
reelevant
retrolistecom
rss api
rss2tg_bot
salesviewerbot
securityheaders
seekport
semrush
sendgrid
sentry
seznam
shortwave image fetcher
site24x7
siteauditbot
siteimprove crawl
sitelock
skroutz imagebot
skroutzbot
skype
slack image proxy
slackbot
snipcart
sogou
spark shipping
splunk
statuscake
stripe
sucuri
swifteq link checker
synthetic bot
taboola
telegram bot
trendiction bot
upday
uptimerobot
uptimia
vaultpress
videootv bot
watchful
webpagetest
webspidermount
wp umbrella
yahoo ad monitoring
yahoo slurp
yahoomailproxy
yandexbot
yeti by naver
yokoy group webhooks
zoominfo

googlebot
googlebot-news
googlebot-image
googlebot-video
storebot-google
googleother
google-inspectiontool
google-extended
adsbot-google-mobile
adsbot-google
mediapartners-google

Other Records