Log In Sign Up

mensaplan.de
robots.txt

Robots Exclusion Standard data for mensaplan.de

Resource Scan

Scan Details

Site Domain mensaplan.de
Base Domain mensaplan.de
Scan Status Ok
Last Scan2024-09-24T16:03:31+00:00
Next Scan 2024-10-24T16:03:31+00:00

Last Scan

Scanned2024-09-24T16:03:31+00:00
URL https://mensaplan.de/robots.txt
Redirect https://www.mensaplan.de/robots.txt
Redirect Domain www.mensaplan.de
Redirect Base mensaplan.de
Domain IPs 217.160.172.247
Redirect IPs 217.160.172.247
Response IP 217.160.172.247
Found Yes
Hash 2e6b29dfe9e37ad31e06b6750a29aba150962b7b9e64a9b32a7badebbfd95927
SimHash 781c6c07e4f7

Groups

*

Rule Path
Disallow /info/
Disallow /de/info/

baiduspider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

daum

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

lcc

Rule Path
Disallow /

linguee

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

checkmarknetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

electricmonk

Rule Path
Disallow /

extlinksbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

obot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

psbot

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

seobility

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

thesubot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

vebidoobot

Rule Path
Disallow /

wget

Rule Path
Disallow /

wotbox

Rule Path
Disallow /
Back to top

Comments

  • Preamble (to human readers)
  • Bots are an important contribution to the Internet ecosystem and provide valuable services to the community.
  • Hence, we appreciate their visit to our website and welcome them.
  • We propose, however, that the following bots (in detail listed below) should not crawl our website:
  • - search engine bots with only little intersection with our target audience
  • - bots that can hardly benefit from crawling our site
  • - bots that have not yet convinced us regarding their usefulness
  • This helps save valuable resources for both them and us.
  • Search engine bots with only little intersection with our target audience:
  • Bots that can hardly benefit from crawling our site:
  • Bots that have not yet convinced us regarding their usefulness:
  • Finally, to all bots out there: please obey Asimov's Laws. Humanity counts on you!
Back to top

Warnings

  • 2 invalid lines.
Back to top