/.well-known/

Log In Sign Up

membros.tiagomiarelli.com.br
robots.txt

Robots Exclusion Standard data for membros.tiagomiarelli.com.br

Archived Snapshots

Resource Scan

Scan Details

Site Domain	membros.tiagomiarelli.com.br
Base Domain	tiagomiarelli.com.br
Scan Status	Failed
Failure Stage	Fetching resource.
Failure Reason	Couldn't connect to server.
Last Scan	2024-09-20T10:26:45+00:00
Next Scan	2024-12-19T10:26:45+00:00

Last Successful Scan

Scanned	2023-09-03T06:25:53+00:00
URL	https://membros.tiagomiarelli.com.br/robots.txt
Domain IPs	184.174.36.112
Response IP	184.174.36.112
Found	Yes
Hash	0d55876220b6429c221afc8431f50c40d1f76d694be8fc8382e1f4aedb9d08c1
SimHash	ec9570d16654

Groups

mauibot

Rule

Path

Disallow

/

blexbot

Rule

Path

Disallow

/

seo spider

Rule

Path

Disallow

/

yandexbot

Rule

Path

Disallow

/

alexawebsearchplatform

Rule

Path

Disallow

/

archive.org_bot

Rule

Path

Disallow

/

baiduspider

Rule

Path

Disallow

/

betabot

Rule

Path

Disallow

/

petalbot

Rule

Path

Disallow

/

dotbot

Rule

Path

Disallow

/

screaming frog seo spider

Rule

Path

Disallow

/

crawl

Rule

Path

Disallow

/

exabot

Rule

Path

Disallow

/

gigabot

Rule

Path

Disallow

/

infoseek sidewinder

Rule

Path

Disallow

/

linkchecker

Rule

Path

Disallow

/

netsongbot

Rule

Path

Disallow

/

dataforseobot

Rule

Path

Disallow

/

dataforseo-bot

Rule

Path

Disallow

/

mj12bot/

Rule

Path

Disallow

/

linkdexbot

Rule

Path

Disallow

/

adsbot

Rule

Path

Disallow

/

awariosmartbot

Rule

Path

Disallow

/

feedspot

Rule

Path

Disallow

/

sogou web spider

Rule

Path

Disallow

/

feedspot/1.0

Rule

Path

Disallow

/

keybot

Rule

Path

Disallow

/

zoominfobot

Rule

Path

Disallow

/

feedspot

Rule

Path

Disallow

/

checkhost

Rule

Path

Disallow

/

orbbot

Rule

Path

Disallow

/

yandeximages

Rule

Path

Disallow

/

wellknownbot

Rule

Path

Disallow

/

surdotlybot

Rule

Path

Disallow

/

gdnplus.com

Rule

Path

Disallow

/

online-webceo-bot

Rule

Path

Disallow

/

scrapy

Rule

Path

Disallow

/

rogerbot

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

dotbot

Rule

Path

Disallow

/

alexibot

Rule

Path

Disallow

/

surveybot

Rule

Path

Disallow

/

xenu

Rule

Path

Disallow

/

exabot

Rule

Path

Disallow

/

gigabot

Rule

Path

Disallow

/

blekkobot

Rule

Path

Disallow

/

mecrawler

Rule

Path

Disallow

/

ia_archiver

Rule

Path

Disallow

/

turnitinbot

Rule

Path

Disallow

/

python-requests

Rule

Path

Disallow

/

python-urllib/2.7

Rule

Path

Disallow

/

curl/7.35.0

Rule

Path

Disallow

/

wp_is_mobile

Rule

Path

Disallow

/

java/11.0.10

Rule

Path

Disallow

/

photon

Rule

Path

Disallow

/

photon/1.0

Rule

Path

Disallow

/

serendeputybot

Rule

Path

Disallow

/

webpage-inspector.com

Rule

Path

Disallow

/

twingly recon

Rule

Path

Disallow

/

bidtellect/0.0.958.0

Rule

Path

Disallow

/

slackbot-linkexpanding 1.0

Rule

Path

Disallow

/

df bot 1.0

Rule

Path

Disallow

/

who.is bot

Rule

Path

Disallow

/

sem rush bot

Rule

Path

Disallow

/

yandex bot

Rule

Path

Disallow

/

sogou bot

Rule

Path

Disallow

/

majestic bot

Rule

Path

Disallow

/

exalead bot

Rule

Path

Disallow

/

baidu bot

Rule

Path

Disallow

/

mediamathbot/1.0

Rule

Path

Disallow

/

trendictionbot0

Rule

Path

Disallow

/

omgili/0.5

Rule

Path

Disallow

/

iframely

Rule

Path

Disallow

/

cortex/1.0

Rule

Path

Disallow

/

trendictionbot0.5.0

Rule

Path

Disallow

/

domains project/1.3.7

Rule

Path

Disallow

/

trendictionbot

Rule

Path

Disallow

/

mediamathbot

Rule

Path

Disallow

/

omgili

Rule

Path

Disallow

/

universalfeedparser/5.2.1

Rule

Path

Disallow

/

woorankreview/2.0

Rule

Path

Disallow

/

feedbot/1.0

Rule

Path

Disallow

/

seekportbot

Rule

Path

Disallow

/

siteauditbot

Rule

Path

Disallow

/

daum/4.1

Rule

Path

Disallow

/

paperlibot/2.1

Rule

Path

Disallow

/

aboutusbot

Rule

Path

Disallow

/

neevabot/1.0

Rule

Path

Disallow

/

feedly/1.0

Rule

Path

Disallow

/

heritrix/3.3.0

Rule

Path

Disallow

/

wget

Rule

Path

Disallow

/

microsoft.url.control

Rule

Path

Disallow

/

ubicrawler

Rule

Path

Disallow

/

icc-crawler

Rule

Path

Disallow

/

sitecheck.internetseer.com

Rule

Path

Disallow

/

zealbot

Rule

Path

Disallow

/

webstripper

Rule

Path

Disallow

/

webcopier

Rule

Path

Disallow

/

httrack

Rule

Path

Disallow

/

libwww

Rule

Path

Disallow

/

baiduspider-image

Rule

Path

Disallow

/

k2spider

Rule

Path

Disallow

/

megaindex.ru

Rule

Path

Disallow

/

megaindex.com

Rule

Path

Disallow

/

linguee bot

Rule

Path

Disallow

/

mappy

Rule

Path

Disallow

/

garlikcrawler

Rule

Path

Disallow

/

feedbot

Rule

Path

Disallow

/

feedlyapp

Rule

Path

Disallow

/

feedly

Rule

Path

Disallow

/

wordpress

Rule

Path

Disallow

/

heritrix

Rule

Path

Disallow

/

fullstorybot

Rule

Path

Disallow

/

ias-va/3.1

Rule

Path

Disallow

/

xpanse-bot

Rule

Path

Disallow

/

phxbot/0.1

Rule

Path

Disallow

/

http://2ip.io

Rule

Path

Disallow

/

semrushbot

Rule

Path

Disallow

/

ahrefsbot

Rule

Path

Disallow

/

mojeebot

Rule

Path

Disallow

/

siteexplorer

Rule

Path

Disallow

/

uptimebot

Rule

Path

Disallow

/

screaming frog seo spider

Rule

Path

Disallow

/

phxbot

Rule

Path

Disallow

/

zgrab/0.x

Rule

Path

Disallow

/

wappalyzer

Rule

Path

Disallow

/

checkmarknetwork/1.0

Rule

Path

Disallow

/

duckduckgo-favicons-bot/1.0;

Rule

Path

Disallow

/

semrushbot

Rule

Path

Disallow

/

gptbot

Rule

Path

Disallow

/

chatgptbot

Rule

Path

Disallow

/

researchscan.comsys.rwth-aachen.de

Rule

Path

Disallow

/

semrushbot/7

Rule

Path

Disallow

/

awariobot/1.0

Rule

Path

Disallow

/

bitsightbot/1.0

Rule

Path

Disallow

/

internetmeasurement/1.0

Rule

Path

Disallow

/

duckduckgo-favicons-bot/1.0

Rule

Path

Disallow

/

yak/1.0

Rule

Path

Disallow

/

uptimerobot/2.0

Rule

Path

Disallow

/

awariorssbot/1.0

Rule

Path

Disallow

/

awariosmartbot/1.0

Rule

Path

Disallow

/

repolookoutbot/v1

Rule

Path

Disallow

/

netcraftsurveyagent/1.0

Rule

Path

Disallow

/

stractbot/0.1

Rule

Path

Disallow

/

yeti/1.1

Rule

Path

Disallow

/

mediatoolkitbot

Rule

Path

Disallow

/

serpstatbot/2.1

Rule

Path

Disallow

/

webwikibot/2.1

Rule

Path

Disallow

/

anderspinkbot/1.1

Rule

Path

Disallow

/

pandalytics/1.0

Rule

Path

Disallow

/

schema-markup-validator

Rule

Path

Disallow

/

anthropic-ai

Rule

Path

Disallow

/

researchscan

Rule

Path

Disallow

/

hstspreload-bot

Rule

Path

Disallow

/

censysinspect/1.1

Rule

Path

Disallow

/

apache-httpclient/4.5.13

Rule

Path

Disallow

/

python/3.10 aiohttp/3.8.5

Rule

Path

Disallow

/

alittle client

Rule

Path

Disallow

/

dalvik/2.1.0

Rule

Path

Disallow

/

python/3.10

Rule

Path

Disallow

/

curl/7.29.0

Rule

Path

Disallow

/

crystal

Rule

Path

Disallow

/

*

Rule

Path

Disallow

/admin/

Disallow

/auth/

Disallow

/assets/browser-update*.js

Disallow

/email/

Disallow

/session

Disallow

/user-api-key

Disallow

/*?api_key*

Disallow

/*?*api_key*

Disallow

/badges

Disallow

/u/

Disallow

/my

Disallow

/search

Disallow

/tag/*/l

Disallow

/g

Disallow

/t/*/*.rss

Disallow

/c/*.rss

googlebot

Rule

Path

Disallow

/admin/

Disallow

/auth/

Disallow

/assets/browser-update*.js

Disallow

/email/

Disallow

/session

Disallow

/user-api-key

Disallow

/*?api_key*

Disallow

/*?*api_key*

Back to top

Other Records

Field

Value

sitemap

https://membros.tiagomiarelli.com.br/sitemap.xml

Back to top

Comments

See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file

Back to top

Warnings

2 invalid lines.

Back to top