wehateporn.com
robots.txt

Groups

grub-client disallow: /
npbot disallow: /
seokicks-robot disallow: /
wget disallow: /
speedy disallow: /
sogou web spider disallow: /
ezooms disallow: /
magpie-crawler disallow: /
yandeximages disallow: /
mj12bot disallow: /
yodaobot disallow: /
nerdbynature.bot disallow: /
discobot disallow: /
knowaboutbot disallow:/
unwindfetchor disallow: /
sitecheck.internetseer.com disallow: /
zealbot disallow: /
msiecrawler disallow: /
webreaper disallow: /
sitesnagger disallow: /
webstripper disallow: /
webcopier disallow: /
fetch disallow: /
offline explorer disallow: /
teleport disallow: /
teleportpro disallow: /
webzip disallow: /
linko disallow: /
httrack disallow: /
microsoft.url.control disallow: /
xenu disallow: /
larbin disallow: /
libwww disallow: /
zyborg disallow: /
download ninja disallow: /

No rules defined. All paths allowed.

Comments

• The grub distributed search engine behaves very badly.
• They totally overwhelm servers with traffic.
• http://www.nameprotect.com/botinfo.html
• Proprietary German backlinks service.
• wget run recusively just *wrecks* server capacity.
• This spider's output isn't public.
• Entireweb
• Foreign-language bot
• Poorly behaved bot
• Brandwatch
• Russian image search engine
• Magestic-12
• Yoydao
• NerdByNature.Net
• Discovery Engine
• No idea.
• Gnip
• These bots are designed to duplicate entire sites.

Warnings

• 1 invalid line.