Log In Sign Up

mvp.rs
robots.txt

Robots Exclusion Standard data for mvp.rs

Resource Scan

Scan Details

Site Domain mvp.rs
Base Domain mvp.rs
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-09-19T02:51:04+00:00
Next Scan 2024-11-18T02:51:04+00:00

Last Successful Scan

Scanned2024-07-22T02:13:02+00:00
URL https://mvp.rs/robots.txt
Domain IPs 116.202.33.97
Response IP 116.202.33.97
Found Yes
Hash eab7f5add8ff4fc78844815baf1e0b2c5322ff08a1f79f6b1d31e19102c47fec
SimHash b37d8f598747

Groups

googlebot

Rule Path
Allow *.js
Allow *.css
Allow *.gif
Allow *.png

*

Rule Path
Disallow /administrator/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Disallow /components/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /layouts/
Disallow /libraries/
Disallow /logs/
Disallow /modules/
Disallow /plugins/
Disallow /tmp/
Disallow /kreiraj-tim/
Disallow /moj-tim/
Disallow /plasman-timova/
Disallow /lista-igraca/
Disallow /timovi/
Disallow /pregled-privatnih-liga
Disallow /moja-liga
Disallow /kreiraj-privatnu-ligu

*

Rule Path
Allow /components/*.js
Allow /components/*.css
Allow /components/*.gif
Allow /components/*.png
Allow /components/*.jpg
Allow /modules/*.js
Allow /modules/*.css
Allow /modules/*.gif
Allow /modules/*.png
Allow /modules/*.jpg
Allow /plugins/*.js
Allow /plugins/*.css
Allow /plugins/*.gif
Allow /plugins/*.png
Allow /plugins/*.jpg

rogerbot
exabot
mj12bot
dotbot
gigabot
ahrefsbot
blackwidow
bot\ [email="craftbot@yahoo.com"]mailto:craftbot@yahoo.com[/email]
chinaclaw
custo
disco
download\ demon
ecatch
eirgrabber
emailsiphon
emailwolf
express\ webpictures
extractorpro
eyenetie
flashget
getright
getweb!
go!zilla
go-ahead-got-it
grabnet
grafula
hmview
httrack
image\ stripper
image\ sucker
indy\ library
interget
internet\ ninja
jetcar
joc\ web\ spider
larbin
leechftp
livelapbot
mass\ downloader
mediatoolkitbot
midown\ tool
mister\ pix
navroad
nearsite
netants
netspider
net\ vampire
netzip
octopus
offline\ explorer
offline\ navigator
pagegrabber
papa\ foto
pavuk
pcbrowser
proximic
realdownload
reget
semrushbot
sitesnagger
smartdownload
superbot
superhttp
surfbot
takeout
teleport\ pro
trendictionbot
voideye
web\ image\ collector
web\ sucker
webauto
webcopier
webfetch
webgo\ is
webleacher
webreaper
websauger
website\ extractor
website\ quester
webstripper
webwhacker
webzip
wget
widow
wwwoffle
xaldon\ webspider
yeti
zeus

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

toutiaospider

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

khtml

Rule Path
Disallow /

spbot

Rule Path
Disallow /

gluten free crawler

Rule Path
Disallow /

titanium octane build

Rule Path
Disallow /

abonti

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

yak

Rule Path
Disallow /

uxcrawlerbot

Rule Path
Disallow /

catexplorador

Rule Path
Disallow /

presto

Rule Path
Disallow /

elinks

Rule Path
Disallow /

comodo ssl checker

Rule Path
Disallow /

masscan

Rule Path
Disallow /

feedlybot

Rule Path
Disallow /

speedyspider

Rule Path
Disallow /

simplepie

Rule Path
Disallow /

cuwhois

Rule Path
Disallow /

spiderman

Rule Path
Disallow /

riddler

Rule Path
Disallow /

coldfusion

Rule Path
Disallow /

esyndicat bot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

www-mechanize

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

hubpages

Rule Path
Disallow /

linqiametadatadownloaderbot

Rule Path
Disallow /

voltron

Rule Path
Disallow /

gimmeusabot

Rule Path
Disallow /

dispatch

Rule Path
Disallow /

xenu

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

lynx

Rule Path
Disallow /

clever internet suite

Rule Path
Disallow /

xml sitemaps generator

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

blogshares spiders

Rule Path
Disallow /

aboutthedomain

Rule Path
Disallow /

c-t bot

Rule Path
Disallow /

awooo

Rule Path
Disallow /

ssearch_bot

Rule Path
Disallow /

steeler

Rule Path
Disallow /

exabot

Rule Path
Disallow /

docomo

Rule Path
Disallow /

imagecoccoc

Rule Path
Disallow /

prlog

Rule Path
Disallow /

testcrawler

Rule Path
Disallow /

econtext classification engine

Rule Path
Disallow /

orangebot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

mnogosearch

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

woobot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

mixrankbot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

k7mlwcbot

Rule Path
Disallow /

telesphoreo

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

pagesinventory

Rule Path
Disallow /

phpcrawl

Rule Path
Disallow /

rome client

Rule Path
Disallow /

siteimprove

Rule Path
Disallow /

faraday

Rule Path
Disallow /

riddler

Rule Path
Disallow /

sitelockspider

Rule Path
Disallow /

grammarly

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

curl

Rule Path
Disallow /

thumbsniper

Rule Path
Disallow /

webspider

Rule Path
Disallow /

rankflex

Rule Path
Disallow /

parsijoo-batch-crawler

Rule Path
Disallow /

mfibot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

cmscrawler

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

hbtools

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

r6_feedfetcher

Rule Path
Disallow /

scanbot

Rule Path
Disallow /

aboutthedomain

Rule Path
Disallow /

crazywebcrawler

Rule Path
Disallow /

css certificate spider

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

gsitecrawler

Rule Path
Disallow /

larbin_2.5.0

Rule Path
Disallow /

mb-sitecrawler

Rule Path
Disallow /

domainsigmacrawler

Rule Path
Disallow /

gluten free crawler

Rule Path
Disallow /

konqueror

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

optimizationcrawler

Rule Path
Disallow /

surdotlybotsysomostestcrawler

Rule Path
Disallow /

roboto

Rule Path
Disallow /

rssingbot scanbot

Rule Path
Disallow /

voltron

Rule Path
Disallow /

wada.vn vietnamese search

Rule Path
Disallow /

mozilla/5.0 jorgee

No rules defined. All paths allowed.

Back to top

Comments

  • If the Joomla site is installed within a folder such as at
  • e.g. www.example.com/joomla/ the robots.txt file MUST be
  • moved to the site root at e.g. www.example.com/robots.txt
  • AND the joomla folder name MUST be prefixed to the disallowed
  • path, e.g. the Disallow rule for the /administrator/ folder
  • MUST be changed to read Disallow: /joomla/administrator/
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/orig.html
  • For syntax checking, see:
  • http://tool.motoricerca.info/robots-checker.phtml
Back to top

Warnings

  • 2 invalid lines.
Back to top