Log In Sign Up

urgolfpro.com
robots.txt

Robots Exclusion Standard data for urgolfpro.com

Resource Scan

Scan Details

Site Domain urgolfpro.com
Base Domain urgolfpro.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-10-16T22:52:25+00:00
Next Scan 2024-11-15T22:52:25+00:00

Last Successful Scan

Scanned2024-08-25T22:21:46+00:00
URL https://urgolfpro.com/robots.txt
Domain IPs 104.21.0.235, 172.67.128.99, 2606:4700:3031::ac43:8063, 2606:4700:3034::6815:eb
Response IP 172.67.128.99
Found Yes
Hash 5de5e6eef0abaf62a0a35cf9577f8b6c5f4e107a5827070a0338eb18b4a991b7
SimHash 639773035ce5

Groups

*

Rule Path
Allow /robots.txt
Disallow /*.js$
Disallow /*.css$
Disallow /*.css?*
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.gif$
Disallow /wp-content/
Disallow /wp-includes/
Disallow /go/

a6-indexer

Rule Path
Disallow /

a6-indexer/1.0

Rule Path
Disallow /

acoon

Rule Path
Disallow /

aghaven

Rule Path
Disallow /

aghaven/nutch-1.2

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

antbot

Rule Path
Disallow /

appcodescrawler

Rule Path
Disallow /

aqua_products

Rule Path
Disallow /

asterias

Rule Path
Disallow /

b2w/0.1

Rule Path
Disallow /

backdoorbot/1.0

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

beetlebot

Rule Path
Disallow /

black hole

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

blowfish/1.0

Rule Path
Disallow /

blp_bbot

Rule Path
Disallow /

bluecoat

Rule Path
Disallow /

bookmark search tool

Rule Path
Disallow /

botalot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

builtbottough

Rule Path
Disallow /

bullseye/1.0

Rule Path
Disallow /

bunnyslippers

Rule Path
Disallow /

butterfly

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

cegbfeieh

Rule Path
Disallow /

centric

Rule Path
Disallow /

centricbot

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

cherrypickerelite/1.0

Rule Path
Disallow /

cherrypickerse/1.0

Rule Path
Disallow /

cis455crawler

Rule Path
Disallow /

citeseerxbot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

compspybot

Rule Path
Disallow /

contextad

Rule Path
Disallow /

contextweb

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

cosmos

Rule Path
Disallow /

covario

Rule Path
Disallow /

covario-ids

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

crescent

Rule Path
Disallow /

cuil

Rule Path
Disallow /

custo

Rule Path
Disallow /

deusu

Rule Path
Disallow /

dibot+java

Rule Path
Disallow /

discobot

Rule Path
Disallow /

dittospyder

Rule Path
Disallow /

dlvr.it

Rule Path
Disallow /

docomo

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dropcatcher.de

Rule Path
Disallow /

educrawler

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

erocrawler

Rule Path
Disallow /

esobisubscriber

Rule Path
Disallow /

exabot

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

ezooms.bot

Rule Path
Disallow /

facebot

Rule Path
Disallow /

fairad client

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

findlinks/2.1.5

Rule Path
Disallow /

flaming attackbot

Rule Path
Disallow /

foobot

Rule Path
Disallow /

fr_org_viewer

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

garlik

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

getright/4.2

Rule Path
Disallow /

gingercrawler

Rule Path
Disallow /

grabber

Rule Path
Disallow /

grub

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

harvest/1.5

Rule Path
Disallow /

hloader

Rule Path
Disallow /

httplib

Rule Path
Disallow /

httrack

Rule Path
Disallow /

humanlinks

Rule Path
Disallow /

hypercrawl

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

ichiro/mobile

Rule Path
Disallow /

ichiro/mobile goo

Rule Path
Disallow /

infonavirobot

Rule Path
Disallow /

infopath

Rule Path
Disallow /

inktomi

Rule Path
Disallow /

inktomisearch

Rule Path
Disallow /

www.integromedb.org/crawler

Rule Path
Disallow /

iron33/1.0.2

Rule Path
Disallow /

itah

Rule Path
Disallow /

jennybot

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

kenjin spider

Rule Path
Disallow /

keyword density/0.9

Rule Path
Disallow /

krowler

Rule Path
Disallow /

larbin

Rule Path
Disallow /

lemurwebcrawler

Rule Path
Disallow /

lexibot

Rule Path
Disallow /

libweb/clshttp

Rule Path
Disallow /

linkscan

Rule Path
Disallow /

linkcheck

Rule Path
Disallow /

linkextractorpro

Rule Path
Disallow /

linkscan

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

lnspiderguy

Rule Path
Disallow /

lssbot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

mata hari

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

megaupload

Rule Path
Disallow /

merchantcentric

Rule Path
Disallow /

merchantcentricbot

Rule Path
Disallow /

microsoft url control

Rule Path
Disallow /

miixpc

Rule Path
Disallow /

minerva

Rule Path
Disallow /

mister pix

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

moget

Rule Path
Disallow /

monsidobot

Rule Path
Disallow /

mozilla

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

msnbot-media

Rule Path
Disallow /

muckrack

Rule Path
Disallow /

netants

Rule Path
Disallow /

netmechanic

Rule Path
Disallow /

nicerspro

Rule Path
Disallow /

nutch

Rule Path
Disallow /

nutraspace

Rule Path
Disallow /

obot

Rule Path
Disallow /

ocelli

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

openbot

Rule Path
Disallow /

openfind

Rule Path
Disallow /

oracle ultra search

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

perman

Rule Path
Disallow /

picsearch

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

propowerbot/2.14

Rule Path
Disallow /

prowebwalker

Rule Path
Disallow /

psbot

Rule Path
Disallow /

ptd-crawler

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

queryn metasearch

Rule Path
Disallow /

radiation retriever 1.1

Rule Path
Disallow /

replazbot

Rule Path
Disallow /

repomonkey

Rule Path
Disallow /

riddler

Rule Path
Disallow /

rma

Rule Path
Disallow /

rssingbot

Rule Path
Disallow /

ru_bot

Rule Path
Disallow /

scooperbot

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

searchpreview

Rule Path
Disallow /

semrush

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

sitecheck

Rule Path
Disallow /

siteimprove

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

slurp

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

soa

Rule Path
Disallow /

soabot

Rule Path
Disallow /

soasearch

Rule Path
Disallow /

sobisubscriber

Rule Path
Disallow /

solomono

Rule Path
Disallow /

solomonobot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

spankbot

Rule Path
Disallow /

spanner

Rule Path
Disallow /

spbot

Rule Path
Disallow /

speedy

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

suzuran

Rule Path
Disallow /

szukacz/1.4

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

telesoft

Rule Path
Disallow /

teoma

Rule Path
Disallow /

the intraformant

Rule Path
Disallow /

thenomad

Rule Path
Disallow /

tighttwatbot

Rule Path
Disallow /

titan

Rule Path
Disallow /

tocrawl/urldispatcher

Rule Path
Disallow /

trident/4.0

Rule Path
Disallow /

true_robot

Rule Path
Disallow /

turingos

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

url control

Rule Path
Disallow /

urldispatcher

Rule Path
Disallow /

url_spider_pro

Rule Path
Disallow /

urly warning

Rule Path
Disallow /

vci

Rule Path
Disallow /

vegebot

Rule Path
Disallow /

visbot

Rule Path
Disallow /

visvo

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

web image collector

Rule Path
Disallow /

webauto

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webenhancer

Rule Path
Disallow /

webmasterworldforumbot

Rule Path
Disallow /

websauger

Rule Path
Disallow /

website quester

Rule Path
Disallow /

webster pro

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webvac

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wesee

Rule Path
Disallow /

wget

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

www-collector-e

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

yyspider

Rule Path
Disallow /

zeus

Rule Path
Disallow /

zibb

Rule Path
Disallow /

zibber

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

toutiaospider

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

friendly_crawler

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

awariobot

Rule Path
Disallow /
Back to top