whofinance.de
robots.txt

Groups

googlebot

google

bingbot

bing

linkedinbot

*

No rules defined. All paths allowed.

Other Records

petalbot
appinsights
semanticscholarbot
dotbot
whatcms
rogerbot
trendictionbot
blexbot
linkfluence
magpie-crawler
mj12bot
mediatoolkitbot
aspiegelbot
domainstatsbot
cincraw
nimbostratus
httrack
serpstatbot
omgili
omgilibot
grapeshotcrawler
megaindex
semanticbot
cocolyzebot
domcopbot
traackr
bomborabot
linguee
webtechbot
domainstatsbot
clickagy
sqlmap
internet-structure-research-project-bot
seekport
awariosmartbot
onalyticabot
buck
riddler
sbl-bot
bvbot
sogou
barkrowler
admantx
adbeat
embed.ly
semantic-visions
voluumdsp
wc-test-dev-bot
gulperbot
mj12bot
crw
petalbot
lucidworks-anda
yandex

baiduspider
yisouspider

bytespider

sogou web spider

sogou inst spider

Other Records

Comments

• allow googlebot and bingbot to crawl all pages
• Enable robots.txt rules for all other crawlers
• Do not crawl links with session IDs, like ?page=
• todo: enable later after institute update
• Disallow: /*?
• SEO-Bots
• User-agent: SemrushBot
• User-agent: AhrefsBot

Warnings

• 1 invalid line.