andrafarm.com
robots.txt

Robots Exclusion Standard data for andrafarm.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	andrafarm.com
Base Domain	andrafarm.com
Scan Status	Failed
Failure Stage	Fetching resource.
Failure Reason	Server returned a client error.
Last Scan	2024-11-01T08:54:51+00:00
Next Scan	2025-01-30T08:54:51+00:00

Last Successful Scan

Scanned	2024-06-12T08:18:59+00:00
URL	https://andrafarm.com/robots.txt
Domain IPs	104.26.4.178, 104.26.5.178, 172.67.75.23, 2606:4700:20::681a:4b2, 2606:4700:20::681a:5b2, 2606:4700:20::ac43:4b17
Response IP	104.26.5.178
Found	Yes
Hash	9e2c2646f5fef2b31c1669cd8a5d63d79d56c3bd6a8fb94032f81e35dea9aff3
SimHash	f2d7128345e6

Groups

googlebot

Rule	Path
Allow	/

Rule

Path

Allow

googlebot-image

Rule	Path
Allow	/

Rule

Path

Allow

googlebot-mobile

Rule	Path
Allow	/

Rule

Path

Allow

mediapartners-google

Rule	Path
Allow	/

Rule

Path

Allow

mediapartners-google*

Rule	Path
Allow	/

Rule

Path

Allow

msnbot

Rule	Path
Allow	/

Rule

Path

Allow

msnbot-media

Rule	Path
Allow	/

Rule

Path

Allow

yahoo! slurp

Rule	Path
Allow	/

Rule

Path

Allow

yahoo!-adcrawler

Rule	Path
Allow	/

Rule

Path

Allow

yahoo-blogs/v3.9

Rule	Path
Allow	/

Rule

Path

Allow

yahoo-mmcrawler

Rule	Path
Allow	/

Rule

Path

Allow

yahooseeker

Rule	Path
Allow	/

Rule

Path

Allow

yahoo-slurp

Rule	Path
Allow	/

Rule

Path

Allow

asterias

Rule	Path
Allow	/

Rule

Path

Allow

becomebot

Rule	Path
Allow	/

Rule

Path

Allow

charlotte

Rule	Path
Allow	/

Rule

Path

Allow

collapsarweb

Rule	Path
Allow	/

Rule

Path

Allow

dolphin

Rule	Path
Allow	/

Rule

Path

Allow

evri

Rule	Path
Allow	/

Rule

Path

Allow

evrinid

Rule	Path
Allow	/

Rule

Path

Allow

exabot

Rule	Path
Allow	/

Rule

Path

Allow

fbot

Rule	Path
Allow	/

Rule

Path

Allow

gigabot

Rule	Path
Allow	/

Rule

Path

Allow

holmes

Rule	Path
Allow	/

Rule

Path

Allow

ia_archiver

Rule	Path
Allow	/

Rule

Path

Allow

irlbot

Rule	Path
Allow	/

Rule

Path

Allow

israbot

Rule	Path
Allow	/

Rule

Path

Allow

linksmanager

Rule	Path
Allow	/

Rule

Path

Allow

linksmanager_bot

Rule	Path
Allow	/

Rule

Path

Allow

naverbot

Rule	Path
Allow	/

Rule

Path

Allow

nutch

Rule	Path
Allow	/

Rule

Path

Allow

onet

Rule	Path
Allow	/

Rule

Path

Allow

onet.pl

Rule	Path
Allow	/

Rule

Path

Allow

onetszukaj

Rule	Path
Allow	/

Rule

Path

Allow

orthogaffe

Rule	Path
Allow	/

Rule

Path

Allow

psbot

Rule	Path
Allow	/

Rule

Path

Allow

robozilla

Rule	Path
Allow	/

Rule

Path

Allow

scrubby

Rule	Path
Allow	/

Rule

Path

Allow

shopwiki

Rule	Path
Allow	/

Rule

Path

Allow

sitecheck.internetseer.com

Rule	Path
Allow	/

Rule

Path

Allow

slurp

Rule	Path
Allow	/

Rule

Path

Allow

speedy

Rule	Path
Allow	/

Rule

Path

Allow

teoma

Rule	Path
Allow	/

Rule

Path

Allow

twiceler

Rule	Path
Allow	/

Rule

Path

Allow

webcrawler

Rule	Path
Allow	/

Rule

Path

Allow

yeti

Rule	Path
Allow	/

Rule

Path

Allow

youdaobot

Rule	Path
Allow	/

Rule

Path

Allow

zyborg

Rule	Path
Allow	/

Rule

Path

Allow

bingbot

Rule

Path

Allow

*

Rule

Path

Disallow

yandex

Rule

Path

Disallow

aipbot

Rule

Path

Disallow

ahrefsbot

Rule

Path

Disallow

alexibot

Rule

Path

Disallow

aqua_products

Rule

Path

Disallow

b2w

Rule

Path

Disallow

backdoorbot

Rule

Path

Disallow

baiduspider

Rule

Path

Disallow

black hole

Rule

Path

Disallow

blowfish

Rule

Path

Disallow

boardtracker

Rule

Path

Disallow

boitho.com-dc

Rule

Path

Disallow

bookmark search tool

Rule

Path

Disallow

botalot

Rule

Path

Disallow

botrighthere

Rule

Path

Disallow

builtbottough

Rule

Path

Disallow

bullseye

Rule

Path

Disallow

bunnyslippers

Rule

Path

Disallow

ccbot

Rule

Path

Disallow

cdlwas_bot

Rule

Path

Disallow

cegbfeieh

Rule

Path

Disallow

cheesebot

Rule

Path

Disallow

cherrypicker

Rule

Path

Disallow

cherrypickerelite

Rule

Path

Disallow

cherrypickerse

Rule

Path

Disallow

converacrawler

Rule

Path

Disallow

copernic

Rule

Path

Disallow

copyrightcheck

Rule

Path

Disallow

cosmos

Rule

Path

Disallow

crawl

Rule

Path

Disallow

crawler

Rule

Path

Disallow

crescent

Rule

Path

Disallow

cuill

Rule

Path

Disallow

disco pump

Rule

Path

Disallow

dittospyder

Rule

Path

Disallow

doc

Rule

Path

Disallow

dotbot

Rule

Path

Disallow

download ninja

Rule

Path

Disallow

dumbot

Rule

Path

Disallow

ellerdalebot

Rule

Path

Disallow

emailcollector

Rule

Path

Disallow

emailsiphon

Rule

Path

Disallow

emailwolf

Rule

Path

Disallow

erocrawler

Rule

Path

Disallow

e-societyrobot

Rule

Path

Disallow

extractorpro

Rule

Path

Disallow

fairad client

Rule

Path

Disallow

fasterfox

Rule

Path

Disallow

fetch

Rule

Path

Disallow

findestars

Rule

Path

Disallow

findlinks

Rule

Path

Disallow

flaming attackbot

Rule

Path

Disallow

foobot

Rule

Path

Disallow

gaisbot

Rule

Path

Disallow

geniebot

Rule

Path

Disallow

getright

Rule

Path

Disallow

grub-client

Rule

Path

Disallow

gurujibot

Rule

Path

Disallow

hailoobot

Rule

Path

Disallow

harvest

Rule

Path

Disallow

hloader

Rule

Path

Disallow

httplib

Rule

Path

Disallow

httrack

Rule

Path

Disallow

humanlinks

Rule

Path

Disallow

ichiro

Rule

Path

Disallow

iconsurf

Rule

Path

Disallow

infonavirobot

Rule

Path

Disallow

iron33

Rule

Path

Disallow

jdxrobot

Rule

Path

Disallow

jennybot

Rule

Path

Disallow

jetsetter

Rule

Path

Disallow

k2spider

Rule

Path

Disallow

kenjin spider

Rule

Path

Disallow

keyword density

Rule

Path

Disallow

larbin

Rule

Path

Disallow

lexibot

Rule

Path

Disallow

libweb

Rule

Path

Disallow

libwww

Rule

Path

Disallow

linguee

Rule

Path

Disallow

linkextractorpro

Rule

Path

Disallow

linko

Rule

Path

Disallow

linkscan

Rule

Path

Disallow

linkwalker

Rule

Path

Disallow

lnspiderguy

Rule

Path

Disallow

localcombot

Rule

Path

Disallow

lwp-trivial

Rule

Path

Disallow

magpie-crawler

Rule

Path

Disallow

mata hari

Rule

Path

Disallow

meltwater

Rule

Path

Disallow

microsoft url control

Rule

Path

Disallow

miixpc

Rule

Path

Disallow

mister pix

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

moget

Rule

Path

Disallow

ms search

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

myonid

Rule

Path

Disallow

netants

Rule

Path

Disallow

netattache

Rule

Path

Disallow

netattache light

Rule

Path

Disallow

netmechanic

Rule

Path

Disallow

nextgensearchbot

Rule

Path

Disallow

nicerspro

Rule

Path

Disallow

nimblecrawler

Rule

Path

Disallow

nlcrawler

Rule

Path

Disallow

npbot

Rule

Path

Disallow

ocelli

Rule

Path

Disallow

offline explorer

Rule

Path

Disallow

omgilibot

Rule

Path

Disallow

omniexplorer_bot

Rule

Path

Disallow

openbot

Rule

Path

Disallow

openfind

Rule

Path

Disallow

openfind data gatherer

Rule

Path

Disallow

oracle ultra search

Rule

Path

Disallow

page-store

Rule

Path

Disallow

paperlibot

Rule

Path

Disallow

peekyou

Rule

Path

Disallow

perman

Rule

Path

Disallow

pipl

Rule

Path

Disallow

piplbot

Rule

Path

Disallow

pompos

Rule

Path

Disallow

postrank

Rule

Path

Disallow

propowerbot

Rule

Path

Disallow

prowebwalker

Rule

Path

Disallow

python-urllib

Rule

Path

Disallow

quepasacreep

Rule

Path

Disallow

queryn metasearch

Rule

Path

Disallow

radiation retriever

Rule

Path

Disallow

rapleaf

Rule

Path

Disallow

repomonkey

Rule

Path

Disallow

rma

Rule

Path

Disallow

scooter

Rule

Path

Disallow

scoutjet

Rule

Path

Disallow

searchpreview

Rule

Path

Disallow

seekbot

Rule

Path

Disallow

sitesnagger

Rule

Path

Disallow

snbot

Rule

Path

Disallow

snitch

Rule

Path

Disallow

sogou spider

Rule

Path

Disallow

spankbot

Rule

Path

Disallow

spanner

Rule

Path

Disallow

spbot

Rule

Path

Disallow

spinn3r

Rule

Path

Disallow

spock

Rule

Path

Disallow

superbot

Rule

Path

Disallow

surveybot

Rule

Path

Disallow

suzuran

Rule

Path

Disallow

szukacz

Rule

Path

Disallow

teleport

Rule

Path

Disallow

teleportpro

Rule

Path

Disallow

telesoft

Rule

Path

Disallow

the intraformant

Rule

Path

Disallow

thenomad

Rule

Path

Disallow

tighttwatbot

Rule

Path

Disallow

titan

Rule

Path

Disallow

tocrawl/urldispatcher

Rule

Path

Disallow

true_robot

Rule

Path

Disallow

turingos

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

tweepz

Rule

Path

Disallow

ubicrawler

Rule

Path

Disallow

url control

Rule

Path

Disallow

url_spider_pro

Rule

Path

Disallow

urly warning

Rule

Path

Disallow

vci

Rule

Path

Disallow

vci webviewer vci webviewer win32

Rule

Path

Disallow

vocusbot

Rule

Path

Disallow

voilabot

Rule

Path

Disallow

voyager

Rule

Path

Disallow

wbsearchbot

Rule

Path

Disallow

web image collector

Rule

Path

Disallow

webauto

Rule

Path

Disallow

webbandit

Rule

Path

Disallow

webcapture

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

webcopy

Rule

Path

Disallow

webenhancer

Rule

Path

Disallow

webmasterworldforumbot

Rule

Path

Disallow

webmirror

Rule

Path

Disallow

webreaper

Rule

Path

Disallow

websauger

Rule

Path

Disallow

website extractor

Rule

Path

Disallow

website quester

Rule

Path

Disallow

webster pro

Rule

Path

Disallow

webstripper

Rule

Path

Disallow

webvac

Rule

Path

Disallow

webzip

Rule

Path

Disallow

wget

Rule

Path

Disallow

winhttrack

Rule

Path

Disallow

wink

Rule

Path

Disallow

wordpress*

Rule

Path

Disallow

www-collector-e

Rule

Path

Disallow

xenu link sleuth

Rule

Path

Disallow

yasni

Rule

Path

Disallow

yodaobot

Rule

Path

Disallow

yoname

Rule

Path

Disallow

yourtraces

Rule

Path

Disallow

zao

Rule

Path

Disallow

zealbot

Rule

Path

Disallow

zeus

Rule

Path

Disallow

zeus link scout

Rule

Path

Disallow

zibber-v0.1

Rule

Path

Disallow

zoominfo

Rule

Path

Disallow

linkdex.com

Rule

Path

Disallow

sogou web spider

Rule

Path

Disallow

sosospider

Rule

Path

Disallow

wotbox

Rule

Path

Disallow

ezooms

Rule

Path

Disallow

seznambot

Rule

Path

Disallow

mlbot

Rule

Path

Disallow

metadatalabs.com

Rule

Path

Disallow

bitlybot

Rule

Path

Disallow

Comments

Selain di atas di stop dulu
Di bawah ini penegasan untuk di stop dulu

Warnings

4 invalid lines.

andrafarm.comrobots.txt

Resource Scan

Scan Details

Last Successful Scan

Groups

googlebot

googlebot-image

googlebot-mobile

mediapartners-google

mediapartners-google*

msnbot

msnbot-media

yahoo! slurp

yahoo!-adcrawler

yahoo-blogs/v3.9

yahoo-mmcrawler

yahooseeker

yahoo-slurp

asterias

becomebot

charlotte

collapsarweb

dolphin

evri

evrinid

exabot

fbot

gigabot

holmes

ia_archiver

irlbot

israbot

linksmanager

linksmanager_bot

naverbot

nutch

onet

onet.pl

onetszukaj

orthogaffe

psbot

robozilla

scrubby

shopwiki

sitecheck.internetseer.com

slurp

speedy

teoma

twiceler

webcrawler

yeti

youdaobot

zyborg

bingbot

*

yandex

aipbot

ahrefsbot

alexibot

aqua_products

b2w

backdoorbot

baiduspider

black hole

blowfish

boardtracker

boitho.com-dc

bookmark search tool

botalot

botrighthere

builtbottough

bullseye

bunnyslippers

ccbot

cdlwas_bot

cegbfeieh

cheesebot

cherrypicker

cherrypickerelite

cherrypickerse

andrafarm.com
robots.txt