old.kreatos.be
robots.txt

Groups

*

Other Records

crawler4j
crawler4j (http://code.google.com/p/crawler4j/)
curious george - www.analyticsseo.com/crawler
curious george - www.analyticsseo.com
megaindex.ru/2.0
megaindex.ru/
mozilla/5.0 (compatible; megaindex.ru/2.0; +https://www.megaindex.ru/?tab=linkanalyze)
mozilla/5.0 (compatible; mj12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
mj12
mozilla/5.0 (compatible; steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
steeler
microsoft-webdav-miniredir/6.1.7601
mozilla/5.0 (compatible; findxbot/1.0; +http://www.findxbot.com)
findxbot
mozilla/5.0 (compatible; seznambot/3.2; +http://fulltext.sblog.cz/)
seznambot
seznam
mozilla/4.0 (compatible; vagabondo/4.0; http://webagent.wise-guys.nl/)
mozilla/4.0 (compatible; vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
vagabondo
jakarta commons-httpclient/3.0.1
lwp::simple/5.822
mozilla/5.0 (compatible; smtbot/1.0; +http://www.similartech.com/smtbot)
smtbot
mozilla/5.0 (compatible; memorybot/1.22.56 +http://internetmemory.org/en/)
memorybot
mozilla/5.0 (compatible; smrjbot/0.0.20)
smrjbot
mozilla/5.0 (compatible; spbot/4.4.2; +http://openlinkprofiler.org/bot )
spbot
mj12bot
baiduspider
exabot
yandex
bspider

Comments

• robots.txt
• This file is to prevent the crawling and indexing of certain parts
• of your site by web crawlers and spiders run by sites like Yahoo!
• and Google. By telling these "robots" where not to go on your site,
• you save bandwidth and server resources.
• This file will be ignored unless it is at the root of your host:
• Used: http://example.com/robots.txt
• Ignored: http://example.com/site/robots.txt
• For more information about the robots.txt standard, see:
• http://www.robotstxt.org/robotstxt.html
• For syntax checking, see:
• http://www.frobee.com/robots-txt-check
• Directories
• Files
• Paths (clean URLs)
• Paths (no clean URLs)
• badbots

Warnings

• 2 invalid lines.