Log In Sign Up

conaf.org.br
robots.txt

Robots Exclusion Standard data for conaf.org.br

Resource Scan

Scan Details

Site Domain conaf.org.br
Base Domain conaf.org.br
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-09-05T14:38:29+00:00
Next Scan 2024-12-04T14:38:29+00:00

Last Successful Scan

Scanned2024-04-16T11:17:05+00:00
URL http://www.conaf.org.br/robots.txt
Domain IPs 179.185.114.150
Response IP 179.185.114.150
Found Yes
Hash afc7f4bd8730e632dfedc38cfe1188e1d432f3a3c057e099b2425cb9a2c2a635
SimHash 627e354a40f0

Groups

*

Rule Path
Disallow /administrator/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Disallow /components/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /layouts/
Disallow /libraries/
Disallow /logs/
Disallow /modules/
Disallow /plugins/
Disallow /tmp/
Disallow /votacaoeletronica/
Disallow /administrator/
Disallow /app/
Disallow /areasegura/
Disallow /bin/
Disallow /cache/
Disallow /cetap/
Disallow /cli/
Disallow /components/
Disallow /images/
Disallow /impostojusto/
Disallow /includes/
Disallow /language/
Disallow /layouts/
Disallow /libraries/
Disallow /licenca-premio/
Disallow /logs/
Disallow /manutencao/
Disallow /media/
Disallow /modules/
Disallow /plugins/
Disallow /repositoriosindifisco/
Disallow /simule-rh/
Disallow /t3-assets/
Disallow /templates/
Disallow /thumbs/
Disallow /tmp/
Disallow /votacaoeletronica/

tlsprober/0.8

Rule Path
Disallow /

scrapy/1.0.3 (+http://scrapy.org)

Rule Path
Disallow /

rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-rogerbot-crawler+shiny@moz.com)

Rule Path
Disallow /

psbot/0.1 (+http://www.picsearch.com/bot.html)

Rule Path
Disallow /

netlyzer fastprobe (see http://netlyzer.com/report/www.vivreshop.com.br for info)

Rule Path
Disallow /

livelapbot/0.2 (http://site.livelap.com/crawler)

Rule Path
Disallow /

go 1.1 package http

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /

mozilla/5.0 (compatible linux x86_64 mail.ru_bot/img/2.0 +http://go.mail.ru/help/robots)

Rule Path
Disallow /

mozilla/5.0 (compatible meanpathbot/1.0 +http://www.meanpath.com/meanpathbot.html)

Rule Path
Disallow /

mozilla/5.0 (compatible yandexbot/3.0 +http://yandex.com/bots)

Rule Path
Disallow /

yandex

Rule Path
Disallow /

deusu

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

python-urllib/2.7

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

moget

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

obot

Rule Path
Disallow /

addthis

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

embedly

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

tweetmemebot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

bdcbot/1.0

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

sunrise

Rule Path
Disallow /

butterfly

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

twengabot-2.0

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

amznkassocbot/4.0

Rule Path
Disallow /

nerdybot

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

mixrankbot

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

riddler

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

swiftbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

psbot

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

hypercrawl

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

netseer

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

alexabot

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

deusu

Rule Path
Disallow /

tdjbot

Rule Path
Disallow /

y!j-asr/0.1 crawler

Rule Path
Disallow /

googlebot

Rule Path
Disallow /dotstore/

msnbot

Rule Path
Disallow /dotstore/

slurp

Rule Path
Disallow /dotstore/

googlebot-image

Rule Path
Disallow /dotstore/

Other Records

Field Value
crawl-delay 60
Back to top

Comments

  • If the Joomla site is installed within a folder
  • eg www.example.com/joomla/ then the robots.txt file
  • MUST be moved to the site root
  • eg www.example.com/robots.txt
  • AND the joomla folder name MUST be prefixed to all of the
  • paths.
  • eg the Disallow rule for the /administrator/ folder MUST
  • be changed to read
  • Disallow: /joomla/administrator/
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/orig.html
  • For syntax checking, see:
  • http://tool.motoricerca.info/robots-checker.phtml
Back to top

Warnings

  • 12 invalid lines.
Back to top