Log In Sign Up

mtbdata.com
robots.txt

Robots Exclusion Standard data for mtbdata.com

Resource Scan

Scan Details

Site Domain mtbdata.com
Base Domain mtbdata.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-08-25T01:34:23+00:00
Next Scan 2024-11-23T01:34:23+00:00

Last Successful Scan

Scanned2024-04-28T00:43:16+00:00
URL https://mtbdata.com/robots.txt
Domain IPs 104.26.4.117, 104.26.5.117, 172.67.71.105, 2606:4700:20::681a:475, 2606:4700:20::681a:575, 2606:4700:20::ac43:4769
Response IP 104.26.4.117
Found Yes
Hash a64296bbb943b8b1eb5ca67ca63b44339005f76e7fcc969a80472a55c7eec911
SimHash a63245daf6c5

Groups

orthogaffe

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

gsa-crawler

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

cncdialer

Rule Path
Disallow /

maxthon

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /BACore/
Disallow /
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.png$
Disallow /*.bmp$
Disallow /*.xml$

seznambot

Rule Path
Disallow /

viglink

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

sputnik

Rule Path
Disallow /

sputnikbot/2.3

Rule Path
Disallow /

curious george

Rule Path
Disallow /

curious george - www.analyticsseo.com

Rule Path
Disallow /

curious george - www.analyticsseo.com/crawler

Rule Path
Disallow /

http://site.ru

Rule Path
Disallow /

site.ru

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

datanyze

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

adscanner

Rule Path
Disallow /

genieo

Rule Path
Disallow /

genieo/1.0

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

zgrab

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

blekkobot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seobility

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

istellabot/1.01.18

Rule Path
Disallow /

istellabot/1.01.18 +http://www.tiscali.it/

Rule Path
Disallow /

istellabot/1.10.2 +http://www.tiscali.it/

Rule Path
Disallow /

mozilla/5.0 (compatible; istellabot/1.01.18 +http://www.tiscali.it/)

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

sogou web spider/4.0

Rule Path
Disallow /

grouphigh

Rule Path
Disallow /

grouphigh/1.0

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

james bot

Rule Path
Disallow /

leikibot

Rule Path
Disallow /

libcurl

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

livelap

Rule Path
Disallow /

lssrocket

Rule Path
Disallow /

magpie

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

gluten free crawler/1.0

Rule Path
Disallow /

serpstatbot/1.0

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

steeler

Rule Path
Disallow /

steeler/3.5

Rule Path
Disallow /

daum

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yeti-mobile

Rule Path
Disallow /

mr.4x3 powered

Rule Path
Disallow /

sjuupbot

Rule Path
Disallow /

viglink

Rule Path
Disallow /

pi-monster

Rule Path
Disallow /

tracemyfile/1.0

Rule Path
Disallow /

xenu's link sleuth 1.1c

Rule Path
Disallow /

obot/2.3.1

Rule Path
Disallow /

cowbot/1.0

Rule Path
Disallow /

deskyobot

Rule Path
Disallow /

deskyobot/1.0

Rule Path
Disallow /

ltx71+-+(http://ltx71.com/)

Rule Path
Disallow /

pandalytics/1.0
ccbot/2.0
surdotlybot
cincraw/1.0
twingly recon-klondike/1.0
yak/1.0
df bot 1.0
crawlson/1.0
ioncrawl
ltx71
woorankreview/2.0

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

filibot/1.0

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

obot/2.3.1

Rule Path
Disallow /

barkrowler/0.9

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /
Back to top

Comments

  • robots.txt
  • Crawlers that are kind enough to obey, but which we'd rather not have
  • unless they're feeding search engines.
  • Some bots are known to be trouble, particularly those designed to copy
  • entire sites. Please obey robots.txt.
  • Sorry, wget in its recursive mode is a frequent problem.
  • Please read the man page and use it properly; there is a
  • --wait option you can use to set the delay between hits,
  • for instance.
  • The 'grub' distributed client has been *very* poorly behaved.
  • Doesn't follow robots.txt anyway, but...
  • Hits many times per second, not acceptable
  • http://www.nameprotect.com/botinfo.html
  • A capture bot, downloads gazillions of pages with no public benefit
  • http://www.webreaper.net/
  • Disallow: /*.css$
  • Disallow: /*.js$
  • User-Agent: aranhabot
  • Crawl-delay: 10
  • Blekkobot
  • Block BlexBot
  • Baiduspider
  • Yandex
  • "Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)"
  • "Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible;Yeti-Mobile/0.1; +http://help.naver.com/robots/)"
Back to top