Log In Sign Up

cryptome.org
robots.txt

Robots Exclusion Standard data for cryptome.org

Resource Scan

Scan Details

Site Domain cryptome.org
Base Domain cryptome.org
Scan Status Ok
Last Scan2024-06-01T03:20:59+00:00
Next Scan 2024-07-01T03:20:59+00:00

Last Scan

Scanned2024-06-01T03:20:59+00:00
URL https://cryptome.org/robots.txt
Domain IPs 209.17.116.160
Response IP 209.17.116.160
Found Yes
Hash 0258527378fdc1aa2e92b44f8a6b3f7131071b889e3fd238de1df40e3add87ed
SimHash 82d542e09f57

Groups

googlebot*

Rule Path
Allow /

ahrefs.com/robot/

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yacy.net/bot

Rule Path
Disallow /

chrome/41.0.2228.0

Rule Path
Disallow /

safari/537.36

Rule Path
Disallow /

safari

Rule Path
Disallow /

applewebkit/537.36

Rule Path
Disallow /

applewebkit

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

applebot

Rule Path
Disallow /

safari/537.36

Rule Path
Disallow /

safari/601.1

Rule Path
Disallow /

safari/

Rule Path
Disallow /

sitelockspider

Rule Path
Disallow /

sitelockspider [en] (winnt; i ;nav)

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

python-requests/

Rule Path
Disallow /

python-requests/2.18.4

Rule Path
Disallow /

sitelockspider/

Rule Path
Disallow /

applewebkit/537.36

Rule Path
Disallow /

mozilla/5.0 (windows nt 5.1) applewebkit/537.36

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

megaindex.ru/

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

transmission/2.42

Rule Path
Disallow /

transmission/2.

Rule Path
Disallow /

transmission/

Rule Path
Disallow /

wget/1.12 (linux-gnu)

Rule Path
Disallow /

wget/1.12

Rule Path
Disallow /

wget/*

Rule Path
Disallow /

cdlwas_bot

Rule Path
Disallow

dillo/2.2

Rule Path
Disallow /

dillo/

Rule Path
Disallow /

pagenest free edition/3.23/

Rule Path
Disallow /

pagenest 3.23/

Rule Path
Disallow /

pagenest/

Rule Path
Disallow /

percolatespiderobot/

Rule Path
Disallow /

backstreet browser/

Rule Path
Disallow /

backstreet browser 3.1/

Rule Path
Disallow /

webvac/

Rule Path
Disallow /

trident/

Rule Path
Disallow /

yandex/

Rule Path
Disallow /

feedtrace-bot/

Rule Path
Disallow /

voyager/

Rule Path
Disallow /

commons-httpclient/

Rule Path
Disallow /

tweetmemebot

Rule Path
Disallow /

python-urllib/

Rule Path
Disallow /

kame-rt

Rule Path
Disallow /

mxbot/

Rule Path
Disallow /

butterfly/

Rule Path
Disallow /

java/

Rule Path
Disallow /

msnbot-newsblogs/

Rule Path
Disallow /

msnbot/

Rule Path
Disallow /

ccbot/

Rule Path
Disallow /

dotbot/

Rule Path
Disallow /

litefinder/

Rule Path
Disallow /

geniebot/

Rule Path
Disallow /

yahoofeedseeker/

Rule Path
Disallow

ask jeeves/teoma

Rule Path
Disallow

nutchcvs

Rule Path
Disallow /

zealbot/

Rule Path
Disallow /

voilabot/

Rule Path
Disallow /

appie/

Rule Path
Disallow /

teoma/

Rule Path
Disallow

msiecrawler/

Rule Path
Disallow /

sun4u/

Rule Path
Disallow /

httrack/

Rule Path
Disallow /

nutch/

Rule Path
Disallow /

sisi/

Rule Path
Disallow /

fyberspider/

Rule Path
Disallow /

girafabot/

Rule Path
Disallow /

lmspider/

Rule Path
Disallow /

np/

Rule Path
Disallow /

robi/

Rule Path
Disallow /

webster pro/

Rule Path
Disallow /

webster/

Rule Path
Disallow /

zeus/

Rule Path
Disallow /

gigabot/

Rule Path
Disallow /

slurp/

Rule Path
Disallow

scirus/

Rule Path
Disallow /

picosearch/

Rule Path
Disallow /

plucker/

Rule Path
Disallow /

disco pump/

Rule Path
Disallow /

gulliver/

Rule Path
Disallow /

vspider/

Rule Path
Disallow /

emailsiphon/

Rule Path
Disallow /

teleport pro/

Rule Path
Disallow /

fetch/

Rule Path
Disallow /

pamuk/

Rule Path
Disallow /

webcopier/

Rule Path
Disallow /

webcapture/

Rule Path
Disallow /

mass downloader/

Rule Path
Disallow /

webcopy/

Rule Path
Disallow /

awv0.8d/

Rule Path
Disallow /

crescent internet toolpak/

Rule Path
Disallow /

joc web spider/

Rule Path
Disallow /

webstripper/

Rule Path
Disallow /

sitesucker/

Rule Path
Disallow /

webdup/

Rule Path
Disallow /

scooter/

Rule Path
Disallow /

python-urllib/

Rule Path
Disallow /

python/

Rule Path
Disallow /

franklin locator/

Rule Path
Disallow /

ck-sillydog/

Rule Path
Disallow /

pockethttp/

Rule Path
Disallow /

voilabot/

Rule Path
Disallow /

xaldon webspider/

Rule Path
Disallow /

webcapture/

Rule Path
Disallow /

webstripper/

Rule Path
Disallow /

webreaper/

Rule Path
Disallow /

teragramwebcrawler/

Rule Path
Disallow /

vagabondo/

Rule Path
Disallow /

nogoop-httpclient/

Rule Path
Disallow /

baiduspider/

Rule Path
Disallow /

w3crobot/

Rule Path
Disallow /

myoperatb/

Rule Path
Disallow /

mj12bot/

Rule Path
Disallow /

wisebot/

Rule Path
Disallow /

exabot/

Rule Path
Disallow /

naverbot/

Rule Path
Disallow /
Back to top

Comments

  • go away
Back to top

Warnings

  • 20 invalid lines.
Back to top