Log In Sign Up

mx1onboard.com
robots.txt

Robots Exclusion Standard data for mx1onboard.com

Resource Scan

Scan Details

Site Domain mx1onboard.com
Base Domain mx1onboard.com
Scan Status Ok
Last Scan2024-09-28T01:37:31+00:00
Next Scan 2024-10-05T01:37:31+00:00

Last Scan

Scanned2024-09-28T01:37:31+00:00
URL https://mx1onboard.com/robots.txt
Domain IPs 2a01:4f9:4a:4744::2, 95.217.204.89
Response IP 95.217.204.89
Found Yes
Hash 9ee9dabd6038cbfb6effa9d655bb7af75dba9ec2acb8923e2851f4235534780a
SimHash a85e1218c833

Groups

*

Rule Path
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/
Disallow /wp-includes/
Allow /feed/$
Disallow /feed
Disallow /comments/feed
Disallow /*/feed/$
Disallow /*/feed/rss/$
Disallow /*/trackback/$
Disallow /*/*/feed/$
Disallow /*/*/feed/rss/$
Disallow /*/*/trackback/$
Disallow /*/*/*/feed/$
Disallow /*/*/*/feed/rss/$
Disallow /*/*/*/trackback/$

compspybot

Rule Path
Disallow /

curious george

Rule Path
Disallow /

cybeye.com

Rule Path
Disallow /

doc

Rule Path
Disallow /

docomo

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

exb language crawler

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

fetch

Rule Path
Disallow /

genieo

Rule Path
Disallow /

genio

Rule Path
Disallow /

gsa-crawler

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

httrack

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

lwnutch

Rule Path
Disallow /

larbin

Rule Path
Disallow /

lexxebot

Rule Path
Disallow /

libwww

Rule Path
Disallow /

linko

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

npbot

Rule Path
Disallow /

openwebindex

Rule Path
Disallow /

rediffnewsbot

Rule Path
Disallow /

seoengworldbot

Rule Path
Disallow /

scanmine

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

wocbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yeti

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

gsa-crawler

Rule Path
Disallow /

libcrawl

Rule Path
Disallow /

linkdex

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

repparser

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

sindice-site-manager

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

woriobot

Rule Path
Disallow /

xenu

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yolinkbot

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wget

Rule Path
Disallow /

zao

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

zyborg

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://mx1onboard.com/sitemap.xml
Back to top

Comments

  • Sitemap
  • Permitimos el feed general para Google Blogsearch.
  • Impedimos que permalink/feed/ sea indexado ya que el
  • feed con los comentarios suele posicionarse en lugar de
  • la entrada y desorienta a los usuarios.
  • Lo mismo con URLs terminadas en /trackback/ que sólo
  • sirven como Trackback URI (y son contenido duplicado).
  • A partir de aquí es opcional pero recomendado.
  • Lista de bots que suelen respetar el robots.txt pero rara
  • vez hacen un buen uso del sitio y abusan bastante…
  • Añadir al gusto del consumidor…
Back to top