go.boarddocs.com robots.txt

*

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	1000

Field

Value

crawl-delay

1000

aboundexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

acapbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

acoon

Rule	Path
Disallow	/

Rule

Path

Disallow

/

admantx

Rule	Path
Disallow	/

Rule

Path

Disallow

/

admedia

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aihitbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

alexibot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aolbot/4.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

applebot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

archive.org

Rule	Path
Disallow	/

Rule

Path

Disallow

/

archive.org_bot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

argus presse

Rule	Path
Disallow	/

Rule

Path

Disallow

/

asterias

Rule	Path
Disallow	/

Rule

Path

Disallow

/

attribot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

backdoorbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

backlinkcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider-image

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider-video

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

binlar_2.6.3

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bitlybot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

black hole

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blowfish/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bot clicker

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bot.araturka.com

Rule	Path
Disallow	/

Rule

Path

Disallow

/

botalot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

brandwatch

Rule	Path
Disallow	/

Rule

Path

Disallow

/

builtbottough

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bullseye

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bullseye/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bunnyslippers

Rule	Path
Disallow	/

Rule

Path

Disallow

/

butterfly/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

buzzbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

buzzsumo

Rule	Path
Disallow	/

Rule

Path

Disallow

/

careerbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ccbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ccbot/2.0 (http://commoncrawl.org/faq/)

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ccbot/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

cegbfeieh

Rule	Path
Disallow	/

Rule

Path

Disallow

/

changedetection

Rule	Path
Disallow	/

Rule

Path

Disallow

/

checkprivacy.or.kr

Rule	Path
Disallow	/

Rule

Path

Disallow

/

cheesebot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

cherrypicker

Rule	Path
Disallow	/

Rule

Path

Disallow

/

cherrypickerelite/1.0

Rule

Path

Disallow

/

cherrypickerse/1.0

Rule

Path

Disallow

/

cis455crawler

Rule

Path

Disallow

/

claritybot

Rule

Path

Disallow

/

claritydailybot

Rule

Path

Disallow

/

clickagy intelligence bot

Rule

Path

Disallow

/

cliqzbot

Rule

Path

Disallow

/

cms crawler

Rule

Path

Disallow

/

coccoc

Rule

Path

Disallow

/

commoncrawler

Rule

Path

Disallow

/

compspybot

Rule

Path

Disallow

/

copyrightcheck

Rule

Path

Disallow

/

cosmos

Rule

Path

Disallow

/

crawler4j (http://code.google.com/p/crawler4j/)

Rule

Path

Disallow

/

crazywebcrawler

Rule

Path

Disallow

/

crazywebcrawler-spider

Rule

Path

Disallow

/

crescent

Rule

Path

Disallow

/

crescent internet toolpak http ole control v.1.0

Rule

Path

Disallow

/

crystalsemantics

Rule

Path

Disallow

/

crystalsemanticsbot

Rule

Path

Disallow

/

dataprovider site explorer

Rule

Path

Disallow

/

dawinci antiplag spider

Rule

Path

Disallow

/

dcpbot/1.2

Rule

Path

Disallow

/

deepcrawl

Rule

Path

Disallow

/

deusu

Rule

Path

Disallow

/

dialogsearch.com

Rule

Path

Disallow

/

diffbot

Rule

Path

Disallow

/

disco pump 3.1

Rule

Path

Disallow

/

discobot

Rule

Path

Disallow

/

doc

Rule

Path

Disallow

/

domain re-animator bot

Rule

Path

Disallow

/

domainappender

Rule

Path

Disallow

/

domainmacrocrawler

Rule

Path

Disallow

/

domainsigmacrawler

Rule

Path

Disallow

/

download ninja

Rule

Path

Disallow

/

ec2linkfinder

Rule

Path

Disallow

/

emailcollector

Rule

Path

Disallow

/

emailsiphon

Rule

Path

Disallow

/

emailwolf

Rule

Path

Disallow

/

erocrawler

Rule

Path

Disallow

/

exabot

Rule

Path

Disallow

/

exabot/3.0

Rule

Path

Disallow

/

exaleadcloudview

Rule

Path

Disallow

/

extractorpro

Rule

Path

Disallow

/

ezooms

Rule

Path

Disallow

/

ezooms robot

Rule

Path

Disallow

/

ezooms.bot

Rule

Path

Disallow

/

ezooms/1.0

Rule

Path

Disallow

/

f-six

Rule

Path

Disallow

/

facebookexternalhit/1.1

Rule

Path

Disallow

/

facebookplatform/1.0

Rule

Path

Disallow

/

fast enterprise crawler

Rule

Path

Disallow

/

favorg

Rule

Path

Disallow

/

feedretriever

Rule

Path

Disallow

/

fetch

Rule

Path

Disallow

/

findxbot

Rule

Path

Disallow

/

flipboardproxy

Rule

Path

Disallow

/

fr_crawler

Rule

Path

Disallow

/

friendica

Rule

Path

Disallow

/

genieo

Rule

Path

Disallow

/

genieo.com

Rule

Path

Disallow

/

gigablastopensource/1.0

Rule

Path

Disallow

/

gluten free crawler

Rule

Path

Disallow

/

gocrawl

Rule

Path

Disallow

/

google-http-java-client

Rule

Path

Disallow

/

grapeshot

Rule

Path

Disallow

/

grapeshotcrawler

Rule

Path

Disallow

/

grepnetstat.com

Rule

Path

Disallow

/

grub-client

Rule

Path

Disallow

/

hailoo news crawler

Rule

Path

Disallow

/

haosouspider

Rule

Path

Disallow

/

harvest/1.5

Rule

Path

Disallow

/

hatena star

Rule

Path

Disallow

/

havij

Rule

Path

Disallow

/

heritrix

Rule

Path

Disallow

/

hloader

Rule

Path

Disallow

/

httplib

Rule

Path

Disallow

/

httrack 3

Rule

Path

Disallow

/

httrack

Rule

Path

Disallow

/

humanlinks

Rule

Path

Disallow

/

ia_archiver

Rule

Path

Disallow

/

ia_archiver-web.archive.org

Rule

Path

Disallow

ibm ica crawler

Rule

Path

Disallow

/

ichiro 3.0

Rule

Path

Disallow

/

ichiro

Rule

Path

Disallow

/

inagist url resolver

Rule

Path

Disallow

/

infonavirobot

Rule

Path

Disallow

/

insitesbot

Rule

Path

Disallow

/

insitesbotisrabot

Rule

Path

Disallow

/

ip-web-crawler.com

Rule

Path

Disallow

/

israbot

Rule

Path

Disallow

/

istellabot

Rule

Path

Disallow

/

istellabot/1.01.18 +http://www.tiscali.it/istellabot/1.10.2 +http://www.tiscali.it/

Rule

Path

Disallow

/

istellabot/1.01.18

Rule

Path

Disallow

/

jack

Rule

Path

Disallow

/

james bot

Rule

Path

Disallow

/

java

Rule

Path

Disallow

/

jennybot

Rule

Path

Disallow

/

js-kit url resolver

Rule

Path

Disallow

/

k2spider

Rule

Path

Disallow

/

kenjin spider

Rule

Path

Disallow

/

komodiabot

Rule

Path

Disallow

/

language-tools.com

Rule

Path

Disallow

/

larbin

Rule

Path

Disallow

/

lemonsources

Rule

Path

Disallow

/

lexibot

Rule

Path

Disallow

/

libweb/clshttp

Rule

Path

Disallow

/

libwww

Rule

Path

Disallow

/

linguee

Rule

Path

Disallow

/

linkdex.com

Rule

Path

Disallow

/

linkdexbot

Rule

Path

Disallow

/

linkedinbot

Rule

Path

Disallow

/

linkextractorpro

Rule

Path

Disallow

/

linkfluence

Rule

Path

Disallow

/

linko

Rule

Path

Disallow

/

linkscan/8.1a unix

Rule

Path

Disallow

/

linkscrawler

Rule

Path

Disallow

/

linkwalker

Rule

Path

Disallow

/

livelapbot

Rule

Path

Disallow

/

ls session

Rule

Path

Disallow

/

lssrocketcrawler

Rule

Path

Disallow

/

ltbot

Rule

Path

Disallow

/

ltx71

Rule

Path

Disallow

/

lwp-trivial

Rule

Path

Disallow

/

lwp-trivial/1.34

Rule

Path

Disallow

/

madaali.de

Rule

Path

Disallow

/

mail.ru

Rule

Path

Disallow

/

mail.ru_bot

Rule

Path

Disallow

/

mail.ru_bot/2.0

Rule

Path

Disallow

/

mata hari

Rule

Path

Disallow

/

maxpointcrawler/nutch-1.1

Rule

Path

Disallow

/

meanpathbot

Rule

Path

Disallow

/

mediapartners-google*

Rule

Path

Disallow

/

megaindex.ru

Rule

Path

Disallow

/

memorybot

Rule

Path

Disallow

/

metauri api/2.0 +metauri.com

Rule

Path

Disallow

/

microsoft data access internet publishing provider protocol discovery

Rule

Path

Disallow

/

microsoft url

Rule

Path

Disallow

/

microsoft-webdav-miniredir/5.1.2600

Rule

Path

Disallow

/

microsoft.url.control

Rule

Path

Disallow

/

miixpc

Rule

Path

Disallow

/

miixpc/4.2

Rule

Path

Disallow

/

mireobot

Rule

Path

Disallow

/

mister pix

Rule

Path

Disallow

/

mixdata.com

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

mj12bot/v1.4.5

Rule

Path

Disallow

/

mlbot

Rule

Path

Disallow

/

moget

Rule

Path

Disallow

/

moget/2.1

Rule

Path

Disallow

/

mozilla/4.0 (cms crawler: http://www.cmscrawler.com)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.01; windows nt; ms search 4.0 robot)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.01; windows nt; ms search 4.0 robot) microsoft

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.01; windows nt; ms search 5.0 robot)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 6.0; windows nt; ms search 4.0 robot)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 6.0; windows nt; ms search 5.0 robot)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 8.0; windows nt 6.0 ; claritydailybot)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; 200pleasebot/1.0; +http://www.200please.com/bot)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; ahrefsbot/5.0; +http://ahrefs.com/robot/)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; aolbot/4.0; +http://www.aol-soft.com/)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; domainappender /1.0; +http://www.profound.net/domainappender)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; domainsigmacrawler/0.1; +http://domainsigma.com/robot)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; exabot/3.0; +http://www.exabot.com/go/robot)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; findxbot/1.0; +http://www.findxbot.com)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; genieo/1.0 http://www.genieo.com/webfilter.html)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; istellabot/1.01.18 +http://www.tiscali.it/)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; mj12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; netseer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; openhosebot/2.1; +http://www.openhose.org/bot.html)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; pad-bot/9.0; +http://www.descargarprogramagratis.com/)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; paperlibot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; semrushbot-si/0.97; +http://www.semrush.com/bot.html)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; sistrix crawler; http://crawler.sistrix.net/)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; softlistbot/2.2; +http://www.softlist.us/)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; tbot/1.0;)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; tweetedtimes bot/1.0; +http://tweetedtimes.com)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; tweetmemebot/3.0; +http://tweetmeme.com/)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; umbot-ln/1.0; mailto: crawling@ubermetrics-technologies.com)

Rule

Path

Disallow

/

mozilla/5.0 (compatible; xovibot/2.0; +http://www.xovibot.net/)

Rule

Path

Disallow

/

mozilla/5.0 (compatible;acapbot/0.1;treat like googlebot)

Rule

Path

Disallow

/

mozilla/5.0 (macintosh; intel mac os x 10.9; rv:28.0) gecko/20100101 firefox/28.0 (flipboardproxy/1.1; +http://flipboard.user-agent: com/browserproxy)

Rule

Path

Disallow

/

mozilla/5.0 (windows nt 6.2) insitesbot/1.0

Rule

Path

Disallow

/

mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.8.1.6) gecko/20070725 firefox/2.0.0.6 - james bot - webcrawler httpuser-agent: ://cognitiveseo.com/bot.html

Rule

Path

Disallow

/

mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.9.1.2) gecko/20090729 firefox/3.5.2 (.net clr 3.5.30729; diffbot/0.user-agent: 1; +http://www.diffbot.com)

Rule

Path

Disallow

/

mozilla/5.0+(x11;+linux+x86_64)+applewebkit/536.5+(khtml,+like+gecko)+chrome/19.0.1084.56+safari/536.5

Rule

Path

Disallow

/

ms search 4.0 robot

Rule

Path

Disallow

/

ms search 5.0 robot

Rule

Path

Disallow

/

msiecrawler

Rule

Path

Disallow

/

mynutchtest

Rule

Path

Disallow

/

nagios

Rule

Path

Disallow

/

nativehost

Rule

Path

Disallow

/

naverbot

Rule

Path

Disallow

/

nerdbynature

Rule

Path

Disallow

/

nerdbynature.bot

Rule

Path

Disallow

/

nerdybot

Rule

Path

Disallow

/

netants

Rule

Path

Disallow

/

netattache

Rule

Path

Disallow

/

netestate

Rule

Path

Disallow

/

netestate ne crawler

Rule

Path

Disallow

/

netmechanic

Rule

Path

Disallow

/

netseer crawler/2.0

Rule

Path

Disallow

/

netseer

Rule

Path

Disallow

/

netseer.com

Rule

Path

Disallow

/

netshelter contentscan

Rule

Path

Disallow

/

newsme/1.0; feedback@news.me

Rule

Path

Disallow

/

ng/2.0

Rule

Path

Disallow

/

nicerspro

Rule

Path

Disallow

/

niki-bot

Rule

Path

Disallow

/

ning/1.0

Rule

Path

Disallow

/

node/simplecrawler 0.3.9 (http://github.com/cgiffard/node-simplecrawler.git)

Rule

Path

Disallow

/

npbot

Rule

Path

Disallow

/

nutch agent

Rule

Path

Disallow

/

obot

Rule

Path

Disallow

/

offline explorer

Rule

Path

Disallow

/

omgilibot

Rule

Path

Disallow

/

openfind data gathere

Rule

Path

Disallow

/

openfind

Rule

Path

Disallow

/

openhosebot

Rule

Path

Disallow

/

openindexdeepspider

Rule

Path

Disallow

/

openindexdeepspider/nutch-1.5-dev

Rule

Path

Disallow

/

openindexshallowspider

Rule

Path

Disallow

/

openindexshallowspider/nutch-1.5-dev

Rule

Path

Disallow

/

openindexspider

Rule

Path

Disallow

/

openindexspider/nutch-1.5-dev

Rule

Path

Disallow

/

orthogaffe

Rule

Path

Disallow

/

pad-bot

Rule

Path

Disallow

/

panscient.com

Rule

Path

Disallow

/

paperlibot

Rule

Path

Disallow

/

pd-crawler

Rule

Path

Disallow

/

peerindex

Rule

Path

Disallow

/

perl lwp

Rule

Path

Disallow

/

photon

Rule

Path

Disallow

/

phpcrawl

Rule

Path

Disallow

/

plukkie

Rule

Path

Disallow

/

postano

Rule

Path

Disallow

/

propowerbot/2.14

Rule

Path

Disallow

/

prowebwalker

Rule

Path

Disallow

/

proximic

Rule

Path

Disallow

/

pulsecrawler/1.1

Rule

Path

Disallow

/

pulsepoint xt3 web scraper

Rule

Path

Disallow

/

python-urllib

Rule

Path

Disallow

/

queryn metasearch

Rule

Path

Disallow

/

queryseekerspider

Rule

Path

Disallow

/

quickobot/quickobot-1

Rule

Path

Disallow

/

quipu

Rule

Path

Disallow

/

raven

Rule

Path

Disallow

/

ravencrawler

Rule

Path

Disallow

/

rbot

Rule

Path

Disallow

/

recorded future

Rule

Path

Disallow

/

repomonkey

Rule

Path

Disallow

/

rexyobot

Rule

Path

Disallow

/

riddler

Rule

Path

Disallow

/

rma

Rule

Path

Disallow

/

rogerbot

Rule

Path

Disallow

/

rogerbotscreaming frog seo spider

Rule

Path

Disallow

/

ruby

Rule

Path

Disallow

/

scrapy

Rule

Path

Disallow

/

scrapybot

Rule

Path

Disallow

/

screaming frog

Rule

Path

Disallow

/

screaming frog seo spider

Rule

Path

Disallow

/

screenerbot

Rule

Path

Disallow

/

searchmetricsbot

Rule

Path

Disallow

/

seexie.com_bot

Rule

Path

Disallow

/

semantic-visions.com crawler

Rule

Path

Disallow

/

semantissimo

Rule

Path

Disallow

/

semrushbot

Rule

Path

Disallow

/

seokicks-robot

Rule

Path

Disallow

/

seolyticscrawler

Rule

Path

Disallow

/

seoprofiler

Rule

Path

Disallow

/

seznambot

Rule

Path

Disallow

/

shopwiki

Rule

Path

Disallow

/

showyoubot (http://showyou.com/crawler)

Rule

Path

Disallow

/

showyoubot

Rule

Path

Disallow

/

simplecrawler

Rule

Path

Disallow

/

sindup

Rule

Path

Disallow

/

sistrix crawler

Rule

Path

Disallow

/

sitebot

Rule

Path

Disallow

/

sitecheck.internetseer.com

Rule

Path

Disallow

/

siteexplorer

Rule

Path

Disallow

/

sitesnagger

Rule

Path

Disallow

/

skimlinks.com

Rule

Path

Disallow

/

smeshbot

Rule

Path

Disallow

/

smtbot

Rule

Path

Disallow

/

socialradarbot

Rule

Path

Disallow

/

softlistbot

Rule

Path

Disallow

/

sogou spider

Rule

Path

Disallow

/

sogou web spider

Rule

Path

Disallow

/

sogou web spider/4.0

Rule

Path

Disallow

/

sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm

Product

Comment

sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm

07)

Rule

Path

Disallow

/

spankbot

Rule

Path

Disallow

/

spanner

Rule

Path

Disallow

/

spbot

Rule

Path

Disallow

/

spbot/3.0

Rule

Path

Disallow

/

speedy spider

Rule

Path

Disallow

/

spiderbot

Rule

Path

Disallow

/

squirrly

Rule

Path

Disallow

/

sqlmap

Rule

Path

Disallow

/

srmse/nutch

Rule

Path

Disallow

/

ssearch_bot

Rule

Path

Disallow

/

stratagems kumo

Rule

Path

Disallow

/

superbot

Rule

Path

Disallow

/

surdotlybot

Rule

Path

Disallow

/

surveybot

Rule

Path

Disallow

/

suzuran

Rule

Path

Disallow

/

swebot

Rule

Path

Disallow

/

szukacz/1.4

Rule

Path

Disallow

/

tbot

Rule

Path

Disallow

/

tbot-nutch

Rule

Path

Disallow

/

tbot/1.0

Rule

Path

Disallow

/

telegrambot

Rule

Path

Disallow

/

teleport

Rule

Path

Disallow

/

teleportpro

Rule

Path

Disallow

/

telesoft

Rule

Path

Disallow

/

the intraformant

Rule

Path

Disallow

/

thenomad

Rule

Path

Disallow

/

tighttwatbot

Rule

Path

Disallow

/

titan

Rule

Path

Disallow

/

tocrawl/urldispatcher

Rule

Path

Disallow

/

trec-kba-bot

Rule

Path

Disallow

/

trendiction

Rule

Path

Disallow

/

trendictionbot

Rule

Path

Disallow

/

true_robot

Rule

Path

Disallow

/

turingos

Rule

Path

Disallow

/

turnitinbot

Rule

Path

Disallow

/

tweetmemebot

Rule

Path

Disallow

/

tweetmemebot/2.11

Rule

Path

Disallow

/

twengabot

Rule

Path

Disallow

/

twitterbot

Rule

Path

Disallow

/

twmbot

Rule

Path

Disallow

/

typhoeus - https://github.com/typhoeus/typhoeus

Rule

Path

Disallow

/

ubermetrics-technologies

Rule

Path

Disallow

/

ubicrawler

Rule

Path

Disallow

/

umbot-ln

Rule

Path

Disallow

/

unisterbot

Rule

Path

Disallow

/

uptimerobot/2.0

Rule

Path

Disallow

/

urlappendbot

Rule

Path

Disallow

/

urly warning

Rule

Path

Disallow

/

vci

Rule

Path

Disallow

/

vci webviewer vci webviewer win32

Rule

Path

Disallow

/

vocusbot

Rule

Path

Disallow

/

voltron

Rule

Path

Disallow

/

voluniabot

Rule

Path

Disallow

/

wada.vn

Rule

Path

Disallow

/

wasalive

Rule

Path

Disallow

/

wbsearchbot

Rule

Path

Disallow

/

web image collector

Rule

Path

Disallow

/

webauto

Rule

Path

Disallow

/

webbandit

Rule

Path

Disallow

/

webbandit/3.50

Rule

Path

Disallow

/

webcopier

Rule

Path

Disallow

/

webcopy

Rule

Path

Disallow

/

webenhancer

Rule

Path

Disallow

/

webindex

Rule

Path

Disallow

/

webmasterworldforumbot

Rule

Path

Disallow

/

webmirror

Rule

Path

Disallow

/

webnl

Rule

Path

Disallow

/

webreaper

Rule

Path

Disallow

/

websauger

Rule

Path

Disallow

/

website extractor

Rule

Path

Disallow

/

website quester

Rule

Path

Disallow

/

webster pro

Rule

Path

Disallow

/

webstripper

Rule

Path

Disallow

/

webstripper/2.02

Rule

Path

Disallow

/

webzip

Rule

Path

Disallow

/

webzip/4.0

Rule

Path

Disallow

/

wesee_bot

Rule

Path

Disallow

/

wget

Rule

Path

Disallow

/

wget/1.5.3

Rule

Path

Disallow

/

wget/1.6

Rule

Path

Disallow

/

wikiwix-bot

Rule

Path

Disallow

/

winhttrack

Rule

Path

Disallow

/

wire/0.22

Rule

Path

Disallow

/

wiseguys robot

Rule

Path

Disallow

/

woobot

Rule

Path

Disallow

/

woriobot

Rule

Path

Disallow

/

worldbrewbot/2.1

Rule

Path

Disallow

/

wotbox

Rule

Path

Disallow

/

wotbox/2.0

Rule

Path

Disallow

/

www-collector-e

Rule

Path

Disallow

/

www.200please.com

Rule

Path

Disallow

/

www.wesee.com

Rule

Path

Disallow

/

www.wise-guys.nl

Rule

Path

Disallow

/

x-cad-se

Rule

Path

Disallow

/

xenu

Rule

Path

Disallow

/

xenu link sleuth

Rule

Path

Disallow

/

xenu's

Rule

Path

Disallow

/

xovibot

Rule

Path

Disallow

/

yacybot

Rule

Path

Disallow

/

yadirectfetcher

Rule

Path

Disallow

/

yahoo pipes 1.0

Rule

Path

Disallow

/

yandex

Rule

Path

Disallow

/

yandexblogs

Rule

Path

Disallow

/

yandexbot

Rule

Path

Disallow

/

yandexbot/3.0

Rule

Path

Disallow

/

yandexcalendar

Rule

Path

Disallow

/

yandexdirectarticular

Rule

Path

Disallow

/

yandexdirectdyn

Rule

Path

Disallow

/

yandeximages

Rule

Path

Disallow

/

yandexmedia

Rule

Path

Disallow

/

yandexmetrika

Rule

Path

Disallow

/

yandexnews

Rule

Path

Disallow

/

yandexpagechecker

Rule

Path

Disallow

/

yeti

Rule

Path

Disallow

/

yioopbot

Rule

Path

Disallow

/

yisouspider

Rule

Path

Disallow

/

youdaobot

Rule

Path

Disallow

/

zao

Rule

Path

Disallow

/

zealbot

Rule

Path

Disallow

/

zeus 32297 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus

Rule

Path

Disallow

/

zoomspider

Rule

Path

Disallow

/

zumbot

Rule

Path

Disallow

/

zyborg

Rule

Path

Disallow

/

msnbot

Rule

Path

Allow

/*/*/Board.nsf/*

Allow

/*/Board.nsf/*

Other Records

Field

Value

crawl-delay

10

slurp

Rule

Path

Allow

/*/*/Board.nsf/*

Allow

/*/Board.nsf/*

Other Records

Field

Value

crawl-delay

10

googlebot

Rule

Path

Allow

/*/*/Board.nsf/*

Allow

/*/Board.nsf/*

Other Records

Field

Value

crawl-delay

10

bingbot

Rule

Path

Allow

/*/*/Board.nsf/*

Allow

/*/Board.nsf/*

Other Records

Field

Value

crawl-delay

10

Site Domain	go.boarddocs.com
Base Domain	boarddocs.com
Scan Status	Ok
Last Scan	2024-10-18T10:28:35+00:00
Next Scan	2024-11-17T10:28:35+00:00

Scanned	2024-10-18T10:28:35+00:00
URL	https://go.boarddocs.com/robots.txt
Domain IPs	142.215.209.151, 216.87.76.76
Response IP	216.87.76.76
Found	Yes
Hash	ceb76361ce05bf41b9c73d3b66821b2f8024bc39df4f72b29dc098c668d09f01
SimHash	3b9301c0c5c0

go.boarddocs.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

Other Records

aboundexbot

acapbot

acoon

admantx

admedia

ahrefsbot

aihitbot

alexibot

aolbot/4.0

applebot

archive.org

archive.org_bot

argus presse

asterias

attribot

backdoorbot/1.0

backlinkcrawler

baiduspider

baiduspider-image

baiduspider-video

baiduspider/2.0

binlar_2.6.3

bitlybot

black hole

blexbot

blowfish/1.0

bot clicker

bot.araturka.com

botalot

brandwatch

builtbottough

bullseye

bullseye/1.0

bunnyslippers

butterfly/1.0

buzzbot

buzzsumo

careerbot

ccbot

ccbot/2.0 (http://commoncrawl.org/faq/)

ccbot/2.0

cegbfeieh

changedetection

checkprivacy.or.kr

cheesebot

cherrypicker

cherrypickerelite/1.0

cherrypickerse/1.0

cis455crawler

claritybot

claritydailybot

clickagy intelligence bot

cliqzbot

cms crawler

coccoc

commoncrawler

compspybot

copyrightcheck

cosmos

crawler4j (http://code.google.com/p/crawler4j/)

crazywebcrawler

crazywebcrawler-spider

crescent

crescent internet toolpak http ole control v.1.0

crystalsemantics

crystalsemanticsbot

dataprovider site explorer

dawinci antiplag spider

dcpbot/1.2

deepcrawl

deusu

dialogsearch.com

diffbot

go.boarddocs.com
robots.txt