dealbert.net
robots.txt

Groups

adsbot-google
adsbot-google-mobile-apps
adidxbot
bingbot
bingpreview
duckduckbot
duckduckgo-favicons-bot
googlebot
googlebot-image
googlebot-mobile
googlebot-news
googlebot-video
mediapartners-google
msnbot
msnbot-media
slurp

*

Comments

• ROBOTS.TXT
• Alphabetically ordered whitelisting of legitimate web robots, which obey the
• Robots Exclusion Standard (robots.txt). Each bot is shortly described in a
• comment above the (list of) user-agent(s). Uncomment or delete bots you do
• not wish to allow on your website / which do not need to visit your website.
• Important: Blank lines are not allowed in the final robots.txt file!
• Updates can be retrieved from: https://github.com/jonasjacek/robots.txt
• This document is licensed with a CC BY-NC-SA 4.0 license.
• Last update: 2017-09-13
• so.com chinese search engine
• User-agent: 360Spider
• User-agent: 360Spider-Image
• User-agent: 360Spider-Video
• google.com landing page quality check
• google.com app resource fetcher
• bing ads bot
• apple.com search engine
• User-agent: Applebot
• user-agent: AppleNewsBot
• baidu.com chinese search engine
• User-agent: Baiduspider
• User-agent: Baiduspider-image
• bing.com international search engine
• commoncrawl.org open repository of web crawl data
• User-agent: CCBot
• cliqz.com german in-product search engine
• User-agent: Cliqzbot
• coccoc.com vietnamese search engine
• User-agent: coccoc
• User-agent: coccocbot-image
• User-agent: coccocbot-web
• daum.net korean search engine
• User-agent: Daumoa
• dazoo.fr french search engine
• User-agent: Dazoobot
• deusu.de german search engine
• User-agent: DeuSu
• duckduckgo.com international privacy search engine
• eurip.com european search engine
• User-agent: EuripBot
• exploratodo.com latin search engine
• User-agent: Exploratodo
• facebook.com social network
• User-agent: Facebot
• feedly.com feed fetcher
• User-agent: Feedly
• findx.com european search engine
• User-agent: Findxbot
• goo.ne.jp japanese search engine
• User-agent: gooblog
• google.com international search engine
• so.com chinese search engine
• User-agent: HaoSouSpider
• goo.ne.jp japanese search engine
• User-agent: ichiro
• istella.it italian search engine
• User-agent: istellabot
• jike.com / chinaso.com chinese search engine
• User-agent: JikeSpider
• lycos.com & hotbot.com international search engine
• User-agent: Lycos
• mail.ru russian search engine
• User-agent: Mail.Ru
• google.com adsense bot
• mojeek.com search engine
• User-agent: MojeekBot
• bing.com international search engine
• orange.com international search engine
• User-agent: OrangeBot
• pinterest.com social networtk
• User-agent: Pinterest
• botje.nl dutch search engine
• User-agent: Plukkie
• qwant.com french search engine
• User-agent: Qwantify
• rambler.ru russian search engine
• User-agent: Rambler
• seznam.cz czech search engine
• User-agent: SeznamBot
• soso.com chinese search engine
• User-agent: Sosospider
• yahoo.com international search engine
• sogou.com chinese search engine
• User-agent: Sogou blog
• User-agent: Sogou inst spider
• User-agent: Sogou News Spider
• User-agent: Sogou Orion spider
• User-agent: Sogou spider2
• User-agent: Sogou web spider
• sputnik.ru russian search engine
• User-agent: SputnikBot
• ask.com international search engine
• User-agent: Teoma
• twitter.com bot
• User-agent: Twitterbot
• wotbox.com international search engine
• User-agent: wotbox
• yacy.net p2p search software
• User-agent: yacybot
• yandex.com russian search engine
• User-agent: Yandex
• User-agent: YandexMobileBot
• search.naver.com south korean search engine
• user-agent: Yeti
• yioop.com international search engine
• User-agent: YioopBot
• yooz.ir iranian search engine
• User-agent: yoozBot
• youdao.com chinese search engine
• User-agent: YoudaoBot
• crawling rule(s)
• disallow all other bots