/.well-known/

Log In Sign Up

goldenstatecollegiatebaseballleague.com
robots.txt

Robots Exclusion Standard data for goldenstatecollegiatebaseballleague.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	goldenstatecollegiatebaseballleague.com
Base Domain	goldenstatecollegiatebaseballleague.com
Scan Status	Failed
Failure Stage	Fetching resource.
Failure Reason	Server returned a client error.
Last Scan	2024-10-09T12:45:37+00:00
Next Scan	2025-01-07T12:45:37+00:00

Last Successful Scan

Scanned	2022-12-19T10:08:46+00:00
URL	http://www.goldenstatecollegiatebaseballleague.com/robots.txt
Domain IPs	34.214.50.254, 44.233.122.200
Response IP	34.214.50.254
Found	Yes
Hash	2f93d9d0c701ec9d6861853bf7db1e5f7b316b4d7f23cbad043d9f23d121b87c
SimHash	539ed34ac3c3

Groups

*

Rule

Path

Disallow

/admin/

Disallow

/search/

Disallow

/img/site_flash/

Disallow

/swf/

Disallow

/css/

Disallow

/cache_css/

Disallow

/js/

Disallow

/cache_js/

Disallow

/files/

Disallow

/theme/

Disallow

/img/

Disallow

/ajax_elements/

Disallow

/view/private/

Disallow

/stats_elements/

Disallow

/users/

Disallow

/news_articles/save_news_hit

a6-indexer

Rule

Path

Disallow

/

aboundex

Rule

Path

Disallow

/

asterias

Rule

Path

Disallow

/

backdoorbot/1.0

Rule

Path

Disallow

/

backlinkcrawler

Rule

Path

Disallow

/

black hole

Rule

Path

Disallow

/

blowfish/1.0

Rule

Path

Disallow

/

botalot

Rule

Path

Disallow

/

builtbottough

Rule

Path

Disallow

/

bullseye/1.0

Rule

Path

Disallow

/

bunnyslippers

Rule

Path

Disallow

/

ca-crawler

Rule

Path

Disallow

/

ccbot

Rule

Path

Disallow

/

ccbot/2.0

Rule

Path

Disallow

/

cegbfeieh

Rule

Path

Disallow

/

cheesebot

Rule

Path

Disallow

/

cherrypicker

Rule

Path

Disallow

/

cherrypickerelite/1.0

Rule

Path

Disallow

/

cherrypickerse/1.0

Rule

Path

Disallow

/

copyrightcheck

Rule

Path

Disallow

/

cosmos

Rule

Path

Disallow

/

crescent

Rule

Path

Disallow

/

crescent internet toolpak http ole control v.1.0

Rule

Path

Disallow

/

dbot

Rule

Path

Disallow

/

dittospyder

Rule

Path

Disallow

/

easouspider

Rule

Path

Disallow

/

eccp

Rule

Path

Disallow

/

emailcollector

Rule

Path

Disallow

/

emailsiphon

Rule

Path

Disallow

/

emailwolf

Rule

Path

Disallow

/

erocrawler

Rule

Path

Disallow

/

exabot/3.0

Rule

Path

Disallow

/

extractorpro

Rule

Path

Disallow

/

foobot

Rule

Path

Disallow

/

fr-crawler

Rule

Path

Disallow

/

friendly crawler

Rule

Path

Disallow

/

gigablastopensource

Rule

Path

Disallow

/

goodzer/2.0

Rule

Path

Disallow

/

grapeshotcrawler/2.0

Rule

Path

Disallow

/

harvest/1.5

Rule

Path

Disallow

/

heritrix/1.14.4

Rule

Path

Disallow

/

hloader

Rule

Path

Disallow

/

httplib

Rule

Path

Disallow

/

hubspot crawler 1.0

Rule

Path

Disallow

/

humanlinks

Rule

Path

Disallow

/

istellabot

Rule

Path

Disallow

/

infonavirobot

Rule

Path

Disallow

/

jennybot

Rule

Path

Disallow

/

kenjin spider

Rule

Path

Disallow

/

keyword density/0.9

Rule

Path

Disallow

/

konqueror/3.5

Rule

Path

Disallow

/

lexibot

Rule

Path

Disallow

/

libweb/clshttp

Rule

Path

Disallow

/

linkextractorpro

Rule

Path

Disallow

/

linkscan/8.1a unix

Rule

Path

Disallow

/

linkwalker

Rule

Path

Disallow

/

lipperhey seo service

Rule

Path

Disallow

/

lnspiderguy

Rule

Path

Disallow

/

lwp-trivial

Rule

Path

Disallow

/

lwp-trivial/1.34

Rule

Path

Disallow

/

mata hari

Rule

Path

Disallow

/

meanpathbot

Rule

Path

Disallow

/

microsoft url control - 5.01.4511

Rule

Path

Disallow

/

microsoft url control - 6.00.8169

Rule

Path

Disallow

/

miixpc

Rule

Path

Disallow

/

miixpc/4.2

Rule

Path

Disallow

/

mister pix

Rule

Path

Disallow

/

mister pix

Rule

Path

Disallow

/

mixrankbot

Rule

Path

Disallow

/

moget

Rule

Path

Disallow

/

moget/2.1

Rule

Path

Disallow

/

mozilla/4

Rule

Path

Disallow

/

mozilla/4.0 (compatible; bullseye; windows 95)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows 95)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows 98)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows nt)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows xp)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows 2000)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows me)

Rule

Path

Disallow

/

mozilla/5

Rule

Path

Disallow

/

nbot/2.0

Rule

Path

Disallow

/

netants

Rule

Path

Disallow

/

netzcheckbot/1.0

Rule

Path

Disallow

/

nicerspro

Rule

Path

Disallow

/

obot/2.3.1

Rule

Path

Disallow

/

offline explorer

Rule

Path

Disallow

/

openfind

Rule

Path

Disallow

/

openfind data gathere

Rule

Path

Disallow

/

pagesinventory

Rule

Path

Disallow

/

panscient.com

Rule

Path

Disallow

/

propowerbot/2.14

Rule

Path

Disallow

/

prowebwalker

Rule

Path

Disallow

/

queryn metasearch

Rule

Path

Disallow

/

repomonkey

Rule

Path

Disallow

/

repomonkey bait & tackle/v1.01

Rule

Path

Disallow

/

riddler

Rule

Path

Disallow

/

rma

Rule

Path

Disallow

/

ru_bot

Rule

Path

Disallow

/

screenerbot crawler beta 2.0

Rule

Path

Disallow

/

semrushbot/0.98~bl

Rule

Path

Disallow

/

seoengworldbot

Rule

Path

Disallow

/

seokicks-robot

Rule

Path

Disallow

/

seplinkbot

Rule

Path

Disallow

/

seplinkbot/1.0

Rule

Path

Disallow

/

seznambot

Rule

Path

Disallow

/

sistrix

Rule

Path

Disallow

/

sitesnagger

Rule

Path

Disallow

/

smtbot/1.0

Rule

Path

Disallow

/

spankbot

Rule

Path

Disallow

/

spbot

Rule

Path

Disallow

/

sogou web spider

Rule

Path

Disallow

/

spanner

Rule

Path

Disallow

/

suzuran

Rule

Path

Disallow

/

szukacz/1.4

Rule

Path

Disallow

/

teleport

Rule

Path

Disallow

/

teleportpro

Rule

Path

Disallow

/

telesoft

Rule

Path

Disallow

/

the intraformant

Rule

Path

Disallow

/

thenomad

Rule

Path

Disallow

/

tighttwatbot

Rule

Path

Disallow

/

titan

Rule

Path

Disallow

/

tocrawl/urldispatcher

Rule

Path

Disallow

/

true_robot

Rule

Path

Disallow

/

true_robot/1.0

Rule

Path

Disallow

/

turingos

Rule

Path

Disallow

/

urlappendbot

Rule

Path

Disallow

/

urly warning

Rule

Path

Disallow

/

vci

Rule

Path

Disallow

/

vci webviewer vci webviewer win32

Rule

Path

Disallow

/

voltron

Rule

Path

Disallow

/

wbsearchbot

Rule

Path

Disallow

/

web image collector

Rule

Path

Disallow

/

webauto

Rule

Path

Disallow

/

webbandit

Rule

Path

Disallow

/

webbandit/3.50

Rule

Path

Disallow

/

webcapture 2.0

Rule

Path

Disallow

/

webcheck 1.10.4

Rule

Path

Disallow

/

webcopier

Rule

Path

Disallow

/

webenhancer

Rule

Path

Disallow

/

webmasterworldforumbot

Rule

Path

Disallow

/

websauger

Rule

Path

Disallow

/

website quester

Rule

Path

Disallow

/

webster pro

Rule

Path

Disallow

/

webstripper

Rule

Path

Disallow

/

webzip

Rule

Path

Disallow

/

webzip/4.0

Rule

Path

Disallow

/

wesee

Rule

Path

Disallow

/

wget

Rule

Path

Disallow

/

wget/1.5.3

Rule

Path

Disallow

/

wget/1.6

Rule

Path

Disallow

/

worldbrewbot/2.1

Rule

Path

Disallow

/

www-collector-e

Rule

Path

Disallow

/

www.integromedb.org/crawler

Rule

Path

Disallow

/

xenu link sleuth

Rule

Path

Disallow

/

xenu's link sleuth 1.1c

Rule

Path

Disallow

/

zeus

Rule

Path

Disallow

/

zeus 32297 webster pro v2.9 win32

Rule

Path

Disallow

/

xovibot

Rule

Path

Disallow

/

zumbot

Rule

Path

Disallow

/

Disallow

/cgi-bin/

ahrefsbot

Rule

Path

Disallow

/

baiduspider
baiduspider
baiduspider+

Rule

Path

Disallow

/

sch-fast-se-crawl02.osl.basefarm.net

Rule

Path

Disallow

/

sch-fast-se-crawl04.osl.basefarm.net

Rule

Path

Disallow

/

yandex

Rule

Path

Disallow

/

moget
ichiro

Rule

Path

Disallow

/

naverbot
yeti

Rule

Path

Disallow

/

baiduspider
baiduspider-video
baiduspider-image

Rule

Path

Disallow

/

sogou spider

Rule

Path

Disallow

/

youdaobot

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

Back to top

Comments

Allow Google, Yahoo
Begin spam-bot disallow.
Begin Exclusion From Directories from robots.txt
disallow OSL.basefarm.net

Back to top

Warnings

2 invalid lines.

Back to top