datadoi.ee
robots.txt

Groups

mediapartners-google*

ubicrawler

doc

zao

sitecheck.internetseer.com

zealbot

msiecrawler

sitesnagger

webstripper

webcopier

fetch

offline explorer

teleport

teleportpro

webzip

linko

httrack

microsoft.url.control

xenu

larbin

libwww

zyborg

download ninja

fast

grub-client

k2spider

npbot

webreaper

facebookexternalhit/1.1

meta-externalagent

meta-externalagent/1.1

amazonbot

gptbot

blexbot

*

Other Records

Comments

• Section for misbehaving bots
• The following directives to block specific robots were borrowed from Wikipedia's robots.txt
• advertising-related bots:
• Crawlers that are kind enough to obey, but which we'd rather not have
• unless they're feeding search engines.
• Some bots are known to be trouble, particularly those designed to copy
• entire sites. Please obey robots.txt.
• Misbehaving: requests much too fast:
• If your DSpace is going down because of someone using recursive wget,
• you can activate the following rule.
• If your own faculty is bringing down your dspace with recursive wget,
• you can advise them to use the --wait option to set the delay between hits.
• User-agent: wget
• Disallow: /
• The 'grub' distributed client has been *very* poorly behaved.
• Doesn't follow robots.txt anyway, but...
• Hits many times per second, not acceptable
• http://www.nameprotect.com/botinfo.html
• A capture bot, downloads gazillions of pages with no public benefit
• http://www.webreaper.net/
• veel facebook

Warnings

• `host` is not a known field.