Log In Sign Up

urbanohio.com
robots.txt

Robots Exclusion Standard data for urbanohio.com

Resource Scan

Scan Details

Site Domain urbanohio.com
Base Domain urbanohio.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-13T11:33:03+00:00
Next Scan 2024-12-12T11:33:03+00:00

Last Successful Scan

Scanned2024-05-17T10:46:06+00:00
URL https://urbanohio.com/robots.txt
Domain IPs 104.21.13.139, 172.67.200.96, 2606:4700:3031::ac43:c860, 2606:4700:3037::6815:d8b
Response IP 172.67.200.96
Found Yes
Hash 6722fc02eef3651992952201e5bd8978f9b58f1d88f165670c8e046bd7962cc3
SimHash 7b016f710bef

Groups

*

Rule Path
Disallow /admin
Disallow /api
Disallow /applications
Disallow /datastore
Disallow /dev
Disallow /oauth
Disallow /plugins
Disallow /system
Disallow /uploads
Disallow /vendor
Disallow /404error.php
Disallow /Credits.txt
Disallow /error.php
Disallow /login
Disallow /logout
Disallow /register
Disallow /lostpassword
Disallow /privacy
Disallow /cookies
Disallow /guidelines
Disallow /terms
Disallow /online
Disallow /staff
Disallow /contact
Disallow /announcement*
Disallow /search
Disallow /discover*
Disallow /rss*
Disallow /tags*
Disallow /activity
Disallow /new-content
Disallow /promote*
Disallow /ourpicks
Disallow /leaderboard
Disallow /pastleaders
Disallow /topmembers
Disallow /*do%3D*
Disallow /*sort%3D*
Disallow /*sortby%3D*
Disallow /*csrf%3D*
Disallow /*csrfKey%3D*
Disallow */?tab=*
Disallow */?_fromLogin=*
Disallow */?_fromLogout=*
Disallow */submit
Disallow */create
Disallow */edit

*

Rule Path
Disallow /startTopic/
Disallow /*?do=add
Disallow /*?do=submit
Disallow /discover/unread/
Disallow /markallread/
Disallow /staff/
Disallow /online/
Disallow /discover/
Disallow /leaderboard/
Disallow /search/
Disallow /*?advancedSearchForm=
Disallow /register/
Disallow /lostpassword/
Disallow /login/
Disallow /*?sortby=
Disallow /*?filter=
Disallow /*?tab=comments
Disallow /*?do=email
Disallow /*?do=findComment
Disallow /*?do=getLastComment
Disallow /*?do=getNewComment
Disallow /*?do=reportComment
Disallow /*?do=markRead
Disallow /profile/
Allow /uploads/monthly_*_*/*

abonti

Rule Path
Disallow /

aboundex

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

acunetix

Rule Path
Disallow /

admantx

Rule Path
Disallow /

afd-verbotsverfahren

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

aibot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

aipbot

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

alligator

Rule Path
Disallow /

allsubmitter

Rule Path
Disallow /

alphabot

Rule Path
Disallow /

anarchie

Rule Path
Disallow /

apexoo

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

aspseek

Rule Path
Disallow /

asterias

Rule Path
Disallow /

attach

Rule Path
Disallow /

autoemailspider

Rule Path
Disallow /

backdoorbot

Rule Path
Disallow /

backlink-ceck

Rule Path
Disallow /

backlink-check

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

backstreet

Rule Path
Disallow /

backweb

Rule Path
Disallow /

badass

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

bandit

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

batchftp

Rule Path
Disallow /

battleztar bazinga

Rule Path
Disallow /

bbbike

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

bdfetch

Rule Path
Disallow /

betabot

Rule Path
Disallow /

bigfoot

Rule Path
Disallow /

bitacle

Rule Path
Disallow /

blackboard

Rule Path
Disallow /

black hole

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

blow

Rule Path
Disallow /

blowfish

Rule Path
Disallow /

boardreader

Rule Path
Disallow /

bolt

Rule Path
Disallow /

botalot

Rule Path
Disallow /

brandprotect

Rule Path
Disallow /

brandwatch

Rule Path
Disallow /

buddy

Rule Path
Disallow /

builtbottough

Rule Path
Disallow /

builtwith

Rule Path
Disallow /

bullseye

Rule Path
Disallow /

bunnyslippers

Rule Path
Disallow /

buzzsumo

Rule Path
Disallow /

calculon

Rule Path
Disallow /

catexplorador

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

cegbfeieh

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

cheteam

Rule Path
Disallow /

chinaclaw

Rule Path
Disallow /

chlooe

Rule Path
Disallow /

claritybot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

cloud mapping

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

cogentbot

Rule Path
Disallow /

cognitiveseo

Rule Path
Disallow /

collector

Rule Path
Disallow /

com.plumanalytics

Rule Path
Disallow /

copier

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

copyscape

Rule Path
Disallow /

cosmos

Rule Path
Disallow /

craftbot

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

crawler.feedback

Rule Path
Disallow /

crawl.sogou.com

Rule Path
Disallow /

crazywebcrawler

Rule Path
Disallow /

crescent

Rule Path
Disallow /

crunchbot

Rule Path
Disallow /

cshttp

Rule Path
Disallow /

curious

Rule Path
Disallow /

custo

Rule Path
Disallow /

databasedrivermysqli

Rule Path
Disallow /

datacha0s

Rule Path
Disallow /

dblbot

Rule Path
Disallow /

demandbase-bot

Rule Path
Disallow /

demon

Rule Path
Disallow /

deusu

Rule Path
Disallow /

devil

Rule Path
Disallow /

digincore

Rule Path
Disallow /

digitalpebble

Rule Path
Disallow /

diibot

Rule Path
Disallow /

dirbuster

Rule Path
Disallow /

disco

Rule Path
Disallow /

discobot

Rule Path
Disallow /

discoverybot

Rule Path
Disallow /

dispatch

Rule Path
Disallow /

dittospyder

Rule Path
Disallow /

dnyzbot

Rule Path
Disallow /

domainappender

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

domainsigmacrawler

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

download wonder

Rule Path
Disallow /

dragonfly

Rule Path
Disallow /

drip

Rule Path
Disallow /

dsearch

Rule Path
Disallow /

dts agent

Rule Path
Disallow /

easydl

Rule Path
Disallow /

ebingbong

Rule Path
Disallow /

ecatch

Rule Path
Disallow /

eccp/1.0

Rule Path
Disallow /

ecxi

Rule Path
Disallow /

eirgrabber

Rule Path
Disallow /

email siphon

Rule Path
Disallow /

email wolf

Rule Path
Disallow /

erocrawler

Rule Path
Disallow /

evc-batch

Rule Path
Disallow /

evil

Rule Path
Disallow /

exabot

Rule Path
Disallow /

express webpictures

Rule Path
Disallow /

extlinksbot

Rule Path
Disallow /

extractor

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

extreme picture finder

Rule Path
Disallow /

eyenetie

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

facebookscraper

Rule Path
Disallow /

fdm

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

fhscan

Rule Path
Disallow /

fimap

Rule Path
Disallow /

firefox/7.0

Rule Path
Disallow /

flashget

Rule Path
Disallow /

flunky

Rule Path
Disallow /

foobot

Rule Path
Disallow /

freeuploader

Rule Path
Disallow /

frontpage

Rule Path
Disallow /

fyberspider

Rule Path
Disallow /

fyrebot

Rule Path
Disallow /

galaxybot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

germcrawler

Rule Path
Disallow /

getintent

Rule Path
Disallow /

getright

Rule Path
Disallow /

getweb

Rule Path
Disallow /

gigablast

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

g-i-g-a-b-o-t

Rule Path
Disallow /

go-ahead-got-it

Rule Path
Disallow /

gotit

Rule Path
Disallow /

gozilla

Rule Path
Disallow /

go!zilla

Rule Path
Disallow /

grabber

Rule Path
Disallow /

grabnet

Rule Path
Disallow /

grafula

Rule Path
Disallow /

grapefx

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

gridbot

Rule Path
Disallow /

gt::www

Rule Path
Disallow /

haansoft

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

harvest

Rule Path
Disallow /

havij

Rule Path
Disallow /

headmasterseo

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

hloader

Rule Path
Disallow /

hmview

Rule Path
Disallow /

htmlparser

Rule Path
Disallow /

http::lite

Rule Path
Disallow /

httrack

Rule Path
Disallow /

humanlinks

Rule Path
Disallow /

hybridbot

Rule Path
Disallow /

iblog

Rule Path
Disallow /

idbot

Rule Path
Disallow /

id-search

Rule Path
Disallow /

ilsebot

Rule Path
Disallow /

image fetch

Rule Path
Disallow /

image sucker

Rule Path
Disallow /

indeedbot

Rule Path
Disallow /

indy library

Rule Path
Disallow /

infonavirobot

Rule Path
Disallow /

infotekies

Rule Path
Disallow /

instabid

Rule Path
Disallow /

intelliseek

Rule Path
Disallow /

interget

Rule Path
Disallow /

internet ninja

Rule Path
Disallow /

internetseer

Rule Path
Disallow /

internetvista monitor

Rule Path
Disallow /

ips-agent

Rule Path
Disallow /

iria

Rule Path
Disallow /

irlbot

Rule Path
Disallow /

iskanie

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

jbrofuzz

Rule Path
Disallow /

jennybot

Rule Path
Disallow /

jetcar

Rule Path
Disallow /

jetty

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

joc web spider

Rule Path
Disallow /

joomla

Rule Path
Disallow /

jorgee

Rule Path
Disallow /

justview

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

kenjin spider

Rule Path
Disallow /

keyword density

Rule Path
Disallow /

kozmosbot

Rule Path
Disallow /

lanshanbot

Rule Path
Disallow /

larbin

Rule Path
Disallow /

leechftp

Rule Path
Disallow /

leechget

Rule Path
Disallow /

lexibot

Rule Path
Disallow /

lftp

Rule Path
Disallow /

libweb

Rule Path
Disallow /

libwhisker

Rule Path
Disallow /

lightspeedsystems

Rule Path
Disallow /

likse

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

linkextractorpro

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

linkscan

Rule Path
Disallow /

linksmanager

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

linqiametadatadownloaderbot

Rule Path
Disallow /

linqiarssbot

Rule Path
Disallow /

linqiascrapebot

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

lipperhey spider

Rule Path
Disallow /

litemage_walker

Rule Path
Disallow /

lmspider

Rule Path
Disallow /

lnspiderguy

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

lwp-request

Rule Path
Disallow /

lwp::simple

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

magnet

Rule Path
Disallow /

mag-net

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

majestic12

Rule Path
Disallow /

majestic seo

Rule Path
Disallow /

majestic-seo

Rule Path
Disallow /

markmonitor

Rule Path
Disallow /

markwatch

Rule Path
Disallow /

masscan

Rule Path
Disallow /

mass downloader

Rule Path
Disallow /

mata hari

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

meanpath bot

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

mediawords

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

metauri

Rule Path
Disallow /

mfc_tear_sample

Rule Path
Disallow /

microsoft data access

Rule Path
Disallow /

microsoft url control

Rule Path
Disallow /

midown tool

Rule Path
Disallow /

miixpc

Rule Path
Disallow /

mister pix

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mojeek

Rule Path
Disallow /

morfeus fucking scanner

Rule Path
Disallow /

mr.4x3

Rule Path
Disallow /

msfrontpage

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

msrabot

Rule Path
Disallow /

ms web services client protocol

Rule Path
Disallow /

muhstik-scan

Rule Path
Disallow /

musobot

Rule Path
Disallow /

name intelligence

Rule Path
Disallow /

nameprotect

Rule Path
Disallow /

navroad

Rule Path
Disallow /

nearsite

Rule Path
Disallow /

needle

Rule Path
Disallow /

nessus

Rule Path
Disallow /

netants

Rule Path
Disallow /

netcraft

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

netlyzer

Rule Path
Disallow /

netmechanic

Rule Path
Disallow /

netspider

Rule Path
Disallow /

nettrack

Rule Path
Disallow /

net vampire

Rule Path
Disallow /

netvibes

Rule Path
Disallow /

netzip

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

nibbler

Rule Path
Disallow /

nicerspro

Rule Path
Disallow /

niki-bot

Rule Path
Disallow /

nikto

Rule Path
Disallow /

nimblecrawler

Rule Path
Disallow /

nimbostratus

Rule Path
Disallow /

ninja

Rule Path
Disallow /

nmap

Rule Path
Disallow /

npbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

obot

Rule Path
Disallow /

octopus

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

offline navigator

Rule Path
Disallow /

oncrawl

Rule Path
Disallow /

openfind

Rule Path
Disallow /

openlinkprofiler

Rule Path
Disallow /

openvas

Rule Path
Disallow /

openvas

Rule Path
Disallow /

orangebot

Rule Path
Disallow /

orangespider

Rule Path
Disallow /

outclicksbot

Rule Path
Disallow /

outfoxbot

Rule Path
Disallow /

pageanalyzer

Rule Path
Disallow /

page analyzer

Rule Path
Disallow /

pagegrabber

Rule Path
Disallow /

page scorer

Rule Path
Disallow /

pagescorer

Rule Path
Disallow /

pandalytics

Rule Path
Disallow /

panscient

Rule Path
Disallow /

papa foto

Rule Path
Disallow /

pavuk

Rule Path
Disallow /

pcbrowser

Rule Path
Disallow /

pecl::http

Rule Path
Disallow /

peoplepal

Rule Path
Disallow /

phpcrawl

Rule Path
Disallow /

picscout

Rule Path
Disallow /

picsearch

Rule Path
Disallow /

picturefinder

Rule Path
Disallow /

pimonster

Rule Path
Disallow /

pi-monster

Rule Path
Disallow /

pixray

Rule Path
Disallow /

pleasecrawl

Rule Path
Disallow /

plumanalytics

Rule Path
Disallow /

pockey

Rule Path
Disallow /

poe-component-client-http

Rule Path
Disallow /

probethenet

Rule Path
Disallow /

propowerbot

Rule Path
Disallow /

prowebwalker

Rule Path
Disallow /

psbot

Rule Path
Disallow /

pump

Rule Path
Disallow /

pxbroker

Rule Path
Disallow /

pycurl

Rule Path
Disallow /

queryn metasearch

Rule Path
Disallow /

quick-crawler

Rule Path
Disallow /

rankactive

Rule Path
Disallow /

rankactivelinkbot

Rule Path
Disallow /

rankflex

Rule Path
Disallow /

rankingbot

Rule Path
Disallow /

rankingbot2

Rule Path
Disallow /

rankivabot

Rule Path
Disallow /

rankurbot

Rule Path
Disallow /

realdownload

Rule Path
Disallow /

reaper

Rule Path
Disallow /

rebelmouse

Rule Path
Disallow /

recorder

Rule Path
Disallow /

redesscrapy

Rule Path
Disallow /

reget

Rule Path
Disallow /

repomonkey

Rule Path
Disallow /

ripper

Rule Path
Disallow /

rocketcrawler

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

s1z.ru

Rule Path
Disallow /

salesintelligent

Rule Path
Disallow /

sbider

Rule Path
Disallow /

scanalert

Rule Path
Disallow /

scanbot

Rule Path
Disallow /

scan.lol

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

screaming

Rule Path
Disallow /

screenerbot

Rule Path
Disallow /

searchestate

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

semrush

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

seolyticscrawler

Rule Path
Disallow /

seomoz

Rule Path
Disallow /

seoprofiler

Rule Path
Disallow /

seoscanners

Rule Path
Disallow /

seositecheckup

Rule Path
Disallow /

seostats

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

sexsearcher

Rule Path
Disallow /

shodan

Rule Path
Disallow /

siphon

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sitebeam

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

siteimprove

Rule Path
Disallow /

sitelockspider

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

sitesucker

Rule Path
Disallow /

site sucker

Rule Path
Disallow /

sitevigil

Rule Path
Disallow /

slysearch

Rule Path
Disallow /

smartdownload

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

snake

Rule Path
Disallow /

snapbot

Rule Path
Disallow /

snoopy

Rule Path
Disallow /

socialrankiobot

Rule Path
Disallow /

sociscraper

Rule Path
Disallow /

sogouspider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

sottopop

Rule Path
Disallow /

spacebison

Rule Path
Disallow /

spammen

Rule Path
Disallow /

spankbot

Rule Path
Disallow /

spanner

Rule Path
Disallow /

sp_auditbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

spyfu

Rule Path
Disallow /

sqlmap

Rule Path
Disallow /

sqlworm

Rule Path
Disallow /

sqworm

Rule Path
Disallow /

steeler

Rule Path
Disallow /

stripper

Rule Path
Disallow /

sucker

Rule Path
Disallow /

sucuri

Rule Path
Disallow /

superbot

Rule Path
Disallow /

superhttp

Rule Path
Disallow /

surfbot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

suzuran

Rule Path
Disallow /

swiftbot

Rule Path
Disallow /

sysscan

Rule Path
Disallow /

szukacz

Rule Path
Disallow /

t0phackteam

Rule Path
Disallow /

t8abot

Rule Path
Disallow /

takeout

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

telesoft

Rule Path
Disallow /

telesphoreo

Rule Path
Disallow /

telesphorep

Rule Path
Disallow /

the intraformant

Rule Path
Disallow /

thenomad

Rule Path
Disallow /

thumbor

Rule Path
Disallow /

tighttwatbot

Rule Path
Disallow /

titan

Rule Path
Disallow /

toata

Rule Path
Disallow /

toweyabot

Rule Path
Disallow /

tracemyfile

Rule Path
Disallow /

trendiction

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

trendiction.com

Rule Path
Disallow /

trendiction.de

Rule Path
Disallow /

true_robot

Rule Path
Disallow /

turingos

Rule Path
Disallow /

turnitin

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

twice

Rule Path
Disallow /

typhoeus

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

upflow

Rule Path
Disallow /

urly warning

Rule Path
Disallow /

urly.warning

Rule Path
Disallow /

vacuum

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

vb project

Rule Path
Disallow /

vci

Rule Path
Disallow /

vericitecrawler

Rule Path
Disallow /

vidiblescraper

Rule Path
Disallow /

virusdie

Rule Path
Disallow /

voideye

Rule Path
Disallow /

voil

Rule Path
Disallow /

voltron

Rule Path
Disallow /

wallpapers/3.0

Rule Path
Disallow /

wallpapershd

Rule Path
Disallow /

wasalive-bot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

webalta

Rule Path
Disallow /

webauto

Rule Path
Disallow /

web auto

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

webcollage

Rule Path
Disallow /

web collage

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webdav

Rule Path
Disallow /

webenhancer

Rule Path
Disallow /

web enhancer

Rule Path
Disallow /

webfetch

Rule Path
Disallow /

web fetch

Rule Path
Disallow /

webfuck

Rule Path
Disallow /

web fuck

Rule Path
Disallow /

webgo is

Rule Path
Disallow /

webimagecollector

Rule Path
Disallow /

webleacher

Rule Path
Disallow /

webmasterworldforumbot

Rule Path
Disallow /

webmeup-crawler

Rule Path
Disallow /

webpix

Rule Path
Disallow /

web pix

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

websauger

Rule Path
Disallow /

web sauger

Rule Path
Disallow /

webshag

Rule Path
Disallow /

websiteextractor

Rule Path
Disallow /

websitequester

Rule Path
Disallow /

website quester

Rule Path
Disallow /

webster

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

websucker

Rule Path
Disallow /

web sucker

Rule Path
Disallow /

webwhacker

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wesee

Rule Path
Disallow /

whack

Rule Path
Disallow /

whacker

Rule Path
Disallow /

whatweb

Rule Path
Disallow /

who.is bot

Rule Path
Disallow /

widow

Rule Path
Disallow /

winhttrack

Rule Path
Disallow /

wiseguys robot

Rule Path
Disallow /

wisenutbot

Rule Path
Disallow /

wonderbot

Rule Path
Disallow /

woobot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

wprecon

Rule Path
Disallow /

wpscan

Rule Path
Disallow /

www-collector-e

Rule Path
Disallow /

www-mechanize

Rule Path
Disallow /

www::mechanize

Rule Path
Disallow /

wwwoffle

Rule Path
Disallow /

x09mozilla

Rule Path
Disallow /

x22mozilla

Rule Path
Disallow /

xaldon webspider

Rule Path
Disallow /

xaldon_webspider

Rule Path
Disallow /

xenu

Rule Path
Disallow /

xpymep1.exe

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

zade

Rule Path
Disallow /

zauba

Rule Path
Disallow /

zauba.io

Rule Path
Disallow /

zermelo

Rule Path
Disallow /

zeus

Rule Path
Disallow /

zgrab

Rule Path
Disallow /

zitebot

Rule Path
Disallow /

zmeu

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

yandexcalendar

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://urbanohio.com.com/sitemap.php
Back to top

Comments

  • Folders and files
  • Authentication
  • Legal information
  • Staff and users
  • Contact page
  • Announcements
  • Content discovery
  • Content promotion
  • Leaderboard
  • With extra parameters
  • Create and edit pages
  • Allow attachments in the uploads folder
  • Block pages with no unique content
  • Block faceted pages and 301 redirect pages
  • Block profile pages as these have little unique value, consume a lot of crawl time and contain hundreds of 301 links
Back to top

Warnings

  • 8 invalid lines.
  • `host` is not a known field.
Back to top