trippreserver.com
robots.txt

Groups

go-http-client/1.1

arquivo-web-crawler
arquivo-web-crawler (compatible; heritrix/1.14.3 +http://arquivo.pt/faq-crawling)

vegebot

panscient.com

mj12bot

easouspider

baiduspider
baiduspider-image
baiduspider-video
baiduspider-news
baiduspider-favo
baiduspider-ads
baiduspider-cpro
baiduspider+
baiduspider+(+http://www.baidu.com/search/spider.htm)
mozilla/5.0 (compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)

icarus6j - (contact: phil@icarus6.com)

icarus6j

voltron

wininet test

*

ia_archiver

duggmirror

teleport

teleportpro

emailcollector

emailsiphon

webbandit

webzip

webreaper

webstripper

web downloader

webcopier

offline explorer pro

httrack website copier

offline commander

leech

websnake

blackwidow

http weazel

googlebot
*

No rules defined. All paths allowed.

Other Records

Comments

• Baiduspider
• Google will skip the wildcard entries in full and only take note of what is actually under the user-agent Googlebot. So it's best to try not to use the useragent Googlebot in robots unless you absolutely have to. And if you do then add all the pages/assets that they need to take note of, even if they are duplicated with the wildcard entries.