Log In Sign Up

friendica.xyz
robots.txt

Robots Exclusion Standard data for friendica.xyz

Resource Scan

Scan Details

Site Domain friendica.xyz
Base Domain friendica.xyz
Scan Status Ok
Last Scan2024-11-14T13:34:45+00:00
Next Scan 2024-11-15T13:34:45+00:00

Last Scan

Scanned2024-11-14T13:34:45+00:00
URL https://friendica.xyz/robots.txt
Domain IPs 185.26.156.246, 2a00:d0c0:200:0:f467:8aff:febd:cc44
Response IP 185.26.156.246
Found Yes
Hash d573e9049f15fab154cc73ce48907e643d4e1e418c2688d6b323ed3d0fb0d066
SimHash f4979f136fdc

Groups

bingbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mediapartners-google*

Rule Path
Disallow /

israbot

Rule Path
Disallow /

orthogaffe

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

fast

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

abcdatos botlink

Rule Path
Disallow /

acme.spider

Rule Path
Disallow /

ahoy! the homepage finder

Rule Path
Disallow /

alkaline

Rule Path
Disallow /

anthill

Rule Path
Disallow /

walhello appie

Rule Path
Disallow /

arachnophilia

Rule Path
Disallow /

arale

Rule Path
Disallow /

araneo

Rule Path
Disallow /

araybot

Rule Path
Disallow /

architextspider

Rule Path
Disallow /

aretha

Rule Path
Disallow /

ariadne

Rule Path
Disallow /

arks

Rule Path
Disallow /

askjeeves

Rule Path
Disallow /

aspider (associative spider)

Rule Path
Disallow /

atn worldwide

Rule Path
Disallow /

atomz.com search robot

Rule Path
Disallow /

auresys

Rule Path
Disallow /

backrub

Rule Path
Disallow /

bay spider

Rule Path
Disallow /

big brother

Rule Path
Disallow /

bjaaland

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

die blinde kuh

Rule Path
Disallow /

bloodhound

Rule Path
Disallow /

borg-bot

Rule Path
Disallow /

boxseabot

Rule Path
Disallow /

bright.net caching robot

Rule Path
Disallow /

bspider

Rule Path
Disallow /

cactvs chemistry spider

Rule Path
Disallow /

calif

Rule Path
Disallow /

cassandra

Rule Path
Disallow /

digimarc marcspider/cgi

Rule Path
Disallow /

checkbot

Rule Path
Disallow /

christcrawler.com

Rule Path
Disallow /

churl

Rule Path
Disallow /

cienciaficcion.net

Rule Path
Disallow /

cmc/0.01

Rule Path
Disallow /

collective

Rule Path
Disallow /

combine system

Rule Path
Disallow /

conceptbot

Rule Path
Disallow /

confuzzledbot

Rule Path
Disallow /

coolbot

Rule Path
Disallow /

web core / roots

Rule Path
Disallow /

xyleme robot

Rule Path
Disallow /

internet cruiser robot

Rule Path
Disallow /

cusco

Rule Path
Disallow /

cyberspyder link test

Rule Path
Disallow /

cydralspider

Rule Path
Disallow /

desert realm spider

Rule Path
Disallow /

deweb(c) katalog/index

Rule Path
Disallow /

dienstspider

Rule Path
Disallow /

digger

Rule Path
Disallow /

digital integrity robot

Rule Path
Disallow /

direct hit grabber

Rule Path
Disallow /

dnabot

Rule Path
Disallow /

download express

Rule Path
Disallow /

dragonbot

Rule Path
Disallow /

dwcp (dridus' web cataloging project)

Rule Path
Disallow /

e-collector

Rule Path
Disallow /

ebiness

Rule Path
Disallow /

eit link verifier robot

Rule Path
Disallow /

elfinbot

Rule Path
Disallow /

emacs-w3 search engine

Rule Path
Disallow /

ananzi

Rule Path
Disallow /

esculapio

Rule Path
Disallow /

esther

Rule Path
Disallow /

evliya celebi

Rule Path
Disallow /

fastcrawler

Rule Path
Disallow /

fluid dynamics search engine robot

Rule Path
Disallow /

felix ide

Rule Path
Disallow /

wild ferret web hopper

Product Comment
wild ferret web hopper 1, #2, #3
Rule Path
Disallow /

fetchrover

Rule Path
Disallow /

fido

Rule Path
Disallow /

hã¤mã¤hã¤kki

Rule Path
Disallow /

kit-fireball

Rule Path
Disallow /

fish search

Rule Path
Disallow /

fouineur

Rule Path
Disallow /

robot francoroute

Rule Path
Disallow /

freecrawl

Rule Path
Disallow /

funnelweb

Rule Path
Disallow /

gammaspider, focusedcrawler

Rule Path
Disallow /

gazz

Rule Path
Disallow /

gcreep

Rule Path
Disallow /

getbot

Rule Path
Disallow /

geturl

Rule Path
Disallow /

golem

Rule Path
Disallow /

googlebot

Rule Path
Disallow /

grapnel/0.01 experiment

Rule Path
Disallow /

griffon

Rule Path
Disallow /

gromit

Rule Path
Disallow /

northern light gulliver

Rule Path
Disallow /

gulper bot

Rule Path
Disallow /

hambot

Rule Path
Disallow /

harvest

Rule Path
Disallow /

havindex

Rule Path
Disallow /

hi (html index) search

Rule Path
Disallow /

hometown spider pro

Rule Path
Disallow /

ht://dig

Rule Path
Disallow /

htmlgobble

Rule Path
Disallow /

hyper-decontextualizer

Rule Path
Disallow /

iajabot

Rule Path
Disallow /

ibm_planetwide

Rule Path
Disallow /

popular iconoclast

Rule Path
Disallow /

ingrid

Rule Path
Disallow /

imagelock

Rule Path
Disallow /

incywincy

Rule Path
Disallow /

informant

Rule Path
Disallow /

infoseek robot 1.0

Rule Path
Disallow /

infoseek sidewinder

Rule Path
Disallow /

infospiders

Rule Path
Disallow /

inspector web

Rule Path
Disallow /

intelliagent

Rule Path
Disallow /

i, robot

Rule Path
Disallow /

iron33

Rule Path
Disallow /

israeli-search

Rule Path
Disallow /

javabee

Rule Path
Disallow /

jbot java web robot

Rule Path
Disallow /

jcrawler

Rule Path
Disallow /

jeeves

Rule Path
Disallow /

jobo java web robot

Rule Path
Disallow /

jobot

Rule Path
Disallow /

joebot

Rule Path
Disallow /

the jubii indexing robot

Rule Path
Disallow /

jumpstation

Rule Path
Disallow /

image.kapsi.net

Rule Path
Disallow /

katipo

Rule Path
Disallow /

kdd-explorer

Rule Path
Disallow /

kilroy

Rule Path
Disallow /

ko_yappo_robot

Rule Path
Disallow /

labelgrabber

Rule Path
Disallow /

larbin

Rule Path
Disallow /

legs

Rule Path
Disallow /

link validator

Rule Path
Disallow /

linkscan

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

lockon

Rule Path
Disallow /

logo.gif crawler

Rule Path
Disallow /

lycos

Rule Path
Disallow /

mac wwwworm

Rule Path
Disallow /

magpie

Rule Path
Disallow /

marvin/infoseek

Rule Path
Disallow /

mattie

Rule Path
Disallow /

mediafox

Rule Path
Disallow /

merzscope

Rule Path
Disallow /

nec-meshexplorer

Rule Path
Disallow /

mindcrawler

Rule Path
Disallow /

mnogosearch search engine software

Rule Path
Disallow /

moget

Rule Path
Disallow /

momspider

Rule Path
Disallow /

monster

Rule Path
Disallow /

motor

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

muncher

Rule Path
Disallow /

muninn

Rule Path
Disallow /

muscat ferret

Rule Path
Disallow /

mwd.search

Rule Path
Disallow /

internet shinchakubin

Rule Path
Disallow /

ndspider

Rule Path
Disallow /

nederland.zoek

Rule Path
Disallow /

netcarta webmap engine

Rule Path
Disallow /

netmechanic

Rule Path
Disallow /

netscoop

Rule Path
Disallow /

newscan-online

Rule Path
Disallow /

nhse web forager

Rule Path
Disallow /

nomad

Rule Path
Disallow /

the northstar robot

Rule Path
Disallow /

nzexplorer

Rule Path
Disallow /

objectssearch

Rule Path
Disallow /

occam

Rule Path
Disallow /

hku www octopus

Rule Path
Disallow /

ontospider

Rule Path
Disallow /

openfind data gatherer

Rule Path
Disallow /

orb search

Rule Path
Disallow /

pack rat

Rule Path
Disallow /

pageboy

Rule Path
Disallow /

parasite

Rule Path
Disallow /

patric

Rule Path
Disallow /

pegasus

Rule Path
Disallow /

the peregrinator

Rule Path
Disallow /

perlcrawler 1.0

Rule Path
Disallow /

phantom

Rule Path
Disallow /

phpdig

Rule Path
Disallow /

piltdownman

Rule Path
Disallow /

pimptrain.com's robot

Rule Path
Disallow /

pioneer

Rule Path
Disallow /

html_analyzer

Rule Path
Disallow /

portal juice spider

Rule Path
Disallow /

pgp key agent

Rule Path
Disallow /

plumtreewebaccessor

Rule Path
Disallow /

poppi

Rule Path
Disallow /

portalb spider

Rule Path
Disallow /

psbot

Rule Path
Disallow /

getterroboplus puu

Rule Path
Disallow /

the python robot

Rule Path
Disallow /

raven search

Rule Path
Disallow /

rbse spider

Rule Path
Disallow /

resume robot

Rule Path
Disallow /

roadhouse crawling system

Rule Path
Disallow /

rixbot

Rule Path
Disallow /

road runner: the imagescape robot

Rule Path
Disallow /

robbie the robot

Rule Path
Disallow /

computingsite robi/1.0

Rule Path
Disallow /

robocrawl spider

Rule Path
Disallow /

robofox

Rule Path
Disallow /

robozilla

Rule Path
Disallow /

roverbot

Rule Path
Disallow /

rules

Rule Path
Disallow /

safetynet robot

Rule Path
Disallow /

scooter

Rule Path
Disallow /

sleek

Rule Path
Disallow /

search.aus-au.com

Rule Path
Disallow /

searchprocess

Rule Path
Disallow /

senrigan

Rule Path
Disallow /

sg-scout

Rule Path
Disallow /

shagseeker

Rule Path
Disallow /

shai'hulud

Rule Path
Disallow /

sift

Rule Path
Disallow /

simmany robot ver1.0

Rule Path
Disallow /

site valet

Rule Path
Disallow /

open text index robot

Rule Path
Disallow /

sitetech-rover

Rule Path
Disallow /

skymob.com

Rule Path
Disallow /

slcrawler

Rule Path
Disallow /

inktomi slurp

Rule Path
Disallow /

smart spider

Rule Path
Disallow /

snooper

Rule Path
Disallow /

solbot

Rule Path
Disallow /

spanner

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

spider_monkey

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

spiderline crawler

Rule Path
Disallow /

spiderman

Rule Path
Disallow /

spiderview(tm)

Rule Path
Disallow /

spry wizard robot

Rule Path
Disallow /

site searcher

Rule Path
Disallow /

suke

Rule Path
Disallow /

suntek search engine

Rule Path
Disallow /

sven

Rule Path
Disallow /

sygol

Rule Path
Disallow /

tach black widow

Rule Path
Disallow /

tarantula

Rule Path
Disallow /

tarspider

Rule Path
Disallow /

tcl w3 robot

Rule Path
Disallow /

techbot

Rule Path
Disallow /

templeton

Rule Path
Disallow /

teomatechnologies

Rule Path
Disallow /

titan

Rule Path
Disallow /

titin

Rule Path
Disallow /

the tkwww robot

Rule Path
Disallow /

tlspider

Rule Path
Disallow /

ucsd crawl

Rule Path
Disallow /

udmsearch

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

url check

Rule Path
Disallow /

url spider pro

Rule Path
Disallow /

valkyrie

Rule Path
Disallow /

verticrawl

Rule Path
Disallow /

victoria

Rule Path
Disallow /

vision-search

Rule Path
Disallow /

void-bot

Rule Path
Disallow /

voyager

Rule Path
Disallow /

vwbot

Rule Path
Disallow /

the nwi robot

Rule Path
Disallow /

w3m2

Rule Path
Disallow /

wallpaper (alias crawlpaper)

Rule Path
Disallow /

the world wide web wanderer

Rule Path
Disallow /

w@pspider by wap4.com

Rule Path
Disallow /

webbandit web spider

Rule Path
Disallow /

webcatcher

Rule Path
Disallow /

webcopy

Rule Path
Disallow /

webfetcher

Rule Path
Disallow /

the webfoot robot

Rule Path
Disallow /

webinator

Rule Path
Disallow /

weblayers

Rule Path
Disallow /

weblinker

Rule Path
Disallow /

webmirror

Rule Path
Disallow /

the web moose

Rule Path
Disallow /

webquest

Rule Path
Disallow /

digimarc marcspider

Rule Path
Disallow /

webs

Rule Path
Disallow /

websnarf

Rule Path
Disallow /

webspider

Rule Path
Disallow /

webvac

Rule Path
Disallow /

webwalk

Rule Path
Disallow /

webwalker

Rule Path
Disallow /

webwatch

Rule Path
Disallow /

wget

Rule Path
Disallow /

whatuseek winona

Rule Path
Disallow /

whowhere robot

Rule Path
Disallow /

wired digital

Rule Path
Disallow /

weblog monitor

Rule Path
Disallow /

w3mir

Rule Path
Disallow /

webstolperer

Rule Path
Disallow /

the web wombat

Rule Path
Disallow /

the world wide web worm

Rule Path
Disallow /

wwwc ver 0.2.5

Rule Path
Disallow /

webzinger

Rule Path
Disallow /

xget

Rule Path
Disallow /

*

Rule Path
Disallow /
Back to top

Comments

  • Taken from https://n.c7.ee/robots.txt
  • Adopted from wikipedia.org/robots.txt
  • Adopted from robotstxt.org/db.html
  • Manual adds: (not fully named) Gbot and Bbot.
  • Essentially 'Bots, get off this land!'. RSS readers consumed by humans should still work.
Back to top