junkowtf.net
robots.txt

Robots Exclusion Standard data for junkowtf.net

Archived Snapshots

Resource Scan

Scan Details

Site Domain	junkowtf.net
Base Domain	junkowtf.net
Scan Status	Ok
Last Scan	5/22/2025, 7:56:59 PM
Next Scan	5/29/2025, 7:56:59 PM

Last Scan

Scanned	5/22/2025, 7:56:59 PM
URL	https://junkowtf.net/robots.txt
Redirect	https://www.junkowtf.net/robots.txt
Redirect Domain	www.junkowtf.net
Redirect Base	junkowtf.net
Domain IPs	104.21.64.249, 172.67.188.56, 2606:4700:3031::ac43:bc38, 2606:4700:3037::6815:40f9
Redirect IPs	104.21.64.249, 172.67.188.56, 2606:4700:3031::ac43:bc38, 2606:4700:3037::6815:40f9
Response IP	104.21.64.249
Found	Yes
Hash	8331df4b55b39e9d0475fda0cfc14eaeded3305c5af7998b58316d2dabf04787
SimHash	63977f835ce5

Groups

*

Rule	Path
Allow	/robots.txt
Disallow	/*.js$
Disallow	/*.css$
Disallow	/.css?
Disallow	/wp-content/
Disallow	/wp-includes/
Disallow	/go/
Disallow	/wp-login.php

Rule

Path

Allow

/robots.txt

Disallow

/*.js$

Disallow

/*.css$

Disallow

/*.css?*

Disallow

/wp-content/

Disallow

/wp-includes/

Disallow

/go/

Disallow

/wp-login.php

a6-indexer

Rule	Path
Disallow	/

Rule

Path

Disallow

a6-indexer/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

acoon

Rule	Path
Disallow	/

Rule

Path

Disallow

aghaven

Rule	Path
Disallow	/

Rule

Path

Disallow

aghaven/nutch-1.2

Rule	Path
Disallow	/

Rule

Path

Disallow

alexibot

Rule	Path
Disallow	/

Rule

Path

Disallow

antbot

Rule	Path
Disallow	/

Rule

Path

Disallow

appcodescrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

aqua_products

Rule	Path
Disallow	/

Rule

Path

Disallow

asterias

Rule	Path
Disallow	/

Rule

Path

Disallow

b2w/0.1

Rule	Path
Disallow	/

Rule

Path

Disallow

backdoorbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

barkrowler

Rule	Path
Disallow	/

Rule

Path

Disallow

beetlebot

Rule	Path
Disallow	/

Rule

Path

Disallow

black hole

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

blowfish/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

blp_bbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bluecoat

Rule	Path
Disallow	/

Rule

Path

Disallow

bookmark search tool

Rule	Path
Disallow	/

Rule

Path

Disallow

botalot

Rule	Path
Disallow	/

Rule

Path

Disallow

bubing

Rule	Path
Disallow	/

Rule

Path

Disallow

builtbottough

Rule	Path
Disallow	/

Rule

Path

Disallow

bullseye/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

bunnyslippers

Rule	Path
Disallow	/

Rule

Path

Disallow

butterfly

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot

Rule	Path
Disallow	/

Rule

Path

Disallow

cegbfeieh

Rule	Path
Disallow	/

Rule

Path

Disallow

centric

Rule	Path
Disallow	/

Rule

Path

Disallow

centricbot

Rule	Path
Disallow	/

Rule

Path

Disallow

cheesebot

Rule	Path
Disallow	/

Rule

Path

Disallow

cherrypicker

Rule	Path
Disallow	/

Rule

Path

Disallow

cherrypickerelite/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

cherrypickerse/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

cis455crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

citeseerxbot

Rule	Path
Disallow	/

Rule

Path

Disallow

cliqzbot

Rule	Path
Disallow	/

Rule

Path

Disallow

compspybot

Rule	Path
Disallow	/

Rule

Path

Disallow

contextad

Rule	Path
Disallow	/

Rule

Path

Disallow

contextweb

Rule	Path
Disallow	/

Rule

Path

Disallow

copyrightcheck

Rule	Path
Disallow	/

Rule

Path

Disallow

cosmos

Rule	Path
Disallow	/

Rule

Path

Disallow

covario

Rule	Path
Disallow	/

Rule

Path

Disallow

covario-ids

Rule	Path
Disallow	/

Rule

Path

Disallow

crazywebcrawler-spider

Rule	Path
Disallow	/

Rule

Path

Disallow

crescent

Rule	Path
Disallow	/

Rule

Path

Disallow

cuil

Rule	Path
Disallow	/

Rule

Path

Disallow

custo

Rule

Path

Disallow

deusu

Rule

Path

Disallow

dibot+java

Rule

Path

Disallow

discobot

Rule

Path

Disallow

dittospyder

Rule

Path

Disallow

dlvr.it

Rule

Path

Disallow

docomo

Rule

Path

Disallow

dotbot

Rule

Path

Disallow

dropcatcher.de

Rule

Path

Disallow

educrawler

Rule

Path

Disallow

emailcollector

Rule

Path

Disallow

emailsiphon

Rule

Path

Disallow

emailwolf

Rule

Path

Disallow

erocrawler

Rule

Path

Disallow

esobisubscriber

Rule

Path

Disallow

exabot

Rule

Path

Disallow

extractorpro

Rule

Path

Disallow

ezooms

Rule

Path

Disallow

ezooms.bot

Rule

Path

Disallow

facebot

Rule

Path

Disallow

fairad client

Rule

Path

Disallow

femtosearchbot

Rule

Path

Disallow

findlinks

Rule

Path

Disallow

findlinks/2.1.5

Rule

Path

Disallow

flaming attackbot

Rule

Path

Disallow

foobot

Rule

Path

Disallow

fr_org_viewer

Rule

Path

Disallow

gaisbot

Rule

Path

Disallow

garlik

Rule

Path

Disallow

garlikcrawler

Rule

Path

Disallow

getright/4.2

Rule

Path

Disallow

gingercrawler

Rule

Path

Disallow

grabber

Rule

Path

Disallow

grub

Rule

Path

Disallow

grub-client

Rule

Path

Disallow

harvest/1.5

Rule

Path

Disallow

hloader

Rule

Path

Disallow

httplib

Rule

Path

Disallow

httrack

Rule

Path

Disallow

humanlinks

Rule

Path

Disallow

hypercrawl

Rule

Path

Disallow

ia_archiver

Rule

Path

Disallow

ichiro

Rule

Path

Disallow

ichiro/mobile

Rule

Path

Disallow

ichiro/mobile goo

Rule

Path

Disallow

infonavirobot

Rule

Path

Disallow

infopath

Rule

Path

Disallow

inktomi

Rule

Path

Disallow

inktomisearch

Rule

Path

Disallow

www.integromedb.org/crawler

Rule

Path

Disallow

iron33/1.0.2

Rule

Path

Disallow

itah

Rule

Path

Disallow

jennybot

Rule

Path

Disallow

jikespider

Rule

Path

Disallow

kenjin spider

Rule

Path

Disallow

keyword density/0.9

Rule

Path

Disallow

krowler

Rule

Path

Disallow

larbin

Rule

Path

Disallow

lemurwebcrawler

Rule

Path

Disallow

lexibot

Rule

Path

Disallow

libweb/clshttp

Rule

Path

Disallow

linkscan

Rule

Path

Disallow

linkcheck

Rule

Path

Disallow

linkextractorpro

Rule

Path

Disallow

linkscan

Rule

Path

Disallow

linkwalker

Rule

Path

Disallow

lnspiderguy

Rule

Path

Disallow

lssbot

Rule

Path

Disallow

ltx71

Rule

Path

Disallow

lwp-trivial

Rule

Path

Disallow

mail.ru

Rule

Path

Disallow

mail.ru_bot

Rule

Path

Disallow

mata hari

Rule

Path

Disallow

mauibot

Rule

Path

Disallow

mbot

Rule

Path

Disallow

megaindex

Rule

Path

Disallow

megaupload

Rule

Path

Disallow

merchantcentric

Rule

Path

Disallow

merchantcentricbot

Rule

Path

Disallow

microsoft url control

Rule

Path

Disallow

miixpc

Rule

Path

Disallow

minerva

Rule

Path

Disallow

mister pix

Rule

Path

Disallow

moget

Rule

Path

Disallow

monsidobot

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

muckrack

Rule

Path

Disallow

netants

Rule

Path

Disallow

netmechanic

Rule

Path

Disallow

nicerspro

Rule

Path

Disallow

nutch

Rule

Path

Disallow

nutraspace

Rule

Path

Disallow

obot

Rule

Path

Disallow

ocelli

Rule

Path

Disallow

offline explorer

Rule

Path

Disallow

openbot

Rule

Path

Disallow

openfind

Rule

Path

Disallow

oracle ultra search

Rule

Path

Disallow

panscient.com

Rule

Path

Disallow

perman

Rule

Path

Disallow

picsearch

Rule

Path

Disallow

piplbot

Rule

Path

Disallow

propowerbot/2.14

Rule

Path

Disallow

prowebwalker

Rule

Path

Disallow

psbot

Rule

Path

Disallow

ptd-crawler

Rule

Path

Disallow

python-urllib

Rule

Path

Disallow

queryn metasearch

Rule

Path

Disallow

radiation retriever 1.1

Rule

Path

Disallow

replazbot

Rule

Path

Disallow

repomonkey

Rule

Path

Disallow

riddler

Rule

Path

Disallow

rma

Rule

Path

Disallow

rssingbot

Rule

Path

Disallow

ru_bot

Rule

Path

Disallow

scooperbot

Rule

Path

Disallow

scoutjet

Rule

Path

Disallow

screaming frog seo spider

Rule

Path

Disallow

searchmetricsbot

Rule

Path

Disallow

searchpreview

Rule

Path

Disallow

seznambot

Rule

Path

Disallow

shopwiki

Rule

Path

Disallow

sistrix

Rule

Path

Disallow

sitebot

Rule

Path

Disallow

sitecheck

Rule

Path

Disallow

siteimprove

Rule

Path

Disallow

sitesnagger

Rule

Path

Disallow

smtbot

Rule

Path

Disallow

soa

Rule

Path

Disallow

soabot

Rule

Path

Disallow

soasearch

Rule

Path

Disallow

sobisubscriber

Rule

Path

Disallow

solomono

Rule

Path

Disallow

solomonobot

Rule

Path

Disallow

sosospider

Rule

Path

Disallow

spankbot

Rule

Path

Disallow

spanner

Rule

Path

Disallow

spbot

Rule

Path

Disallow

speedy

Rule

Path

Disallow

sputnikbot

Rule

Path

Disallow

suzuran

Rule

Path

Disallow

szukacz/1.4

Rule

Path

Disallow

teleport

Rule

Path

Disallow

teleportpro

Rule

Path

Disallow

telesoft

Rule

Path

Disallow

teoma

Rule

Path

Disallow

the intraformant

Rule

Path

Disallow

thenomad

Rule

Path

Disallow

tighttwatbot

Rule

Path

Disallow

titan

Rule

Path

Disallow

tocrawl/urldispatcher

Rule

Path

Disallow

trident/4.0

Rule

Path

Disallow

true_robot

Rule

Path

Disallow

turingos

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

twiceler

Rule

Path

Disallow

url control

Rule

Path

Disallow

urldispatcher

Rule

Path

Disallow

url_spider_pro

Rule

Path

Disallow

urly warning

Rule

Path

Disallow

vci

Rule

Path

Disallow

vegebot

Rule

Path

Disallow

visbot

Rule

Path

Disallow

visvo

Rule

Path

Disallow

voilabot

Rule

Path

Disallow

web image collector

Rule

Path

Disallow

webauto

Rule

Path

Disallow

webbandit

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

webenhancer

Rule

Path

Disallow

webmasterworldforumbot

Rule

Path

Disallow

websauger

Rule

Path

Disallow

website quester

Rule

Path

Disallow

webster pro

Rule

Path

Disallow

webstripper

Rule

Path

Disallow

webvac

Rule

Path

Disallow

webzip

Rule

Path

Disallow

wesee

Rule

Path

Disallow

wget

Rule

Path

Disallow

wotbox

Rule

Path

Disallow

www-collector-e

Rule

Path

Disallow

xovibot

Rule

Path

Disallow

yacybot

Rule

Path

Disallow

yodaobot

Rule

Path

Disallow

youdaobot

Rule

Path

Disallow

yyspider

Rule

Path

Disallow

zeus

Rule

Path

Disallow

zibb

Rule

Path

Disallow

zibber

Rule

Path

Disallow

zumbot

Rule

Path

Disallow

amazonbot

Rule

Path

Disallow

friendly_crawler

Rule

Path

Disallow

imagesiftbot

Rule

Path

Disallow

awariobot

Rule

Path

Disallow

junkowtf.netrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

a6-indexer

a6-indexer/1.0

acoon

aghaven

aghaven/nutch-1.2

alexibot

antbot

appcodescrawler

aqua_products

asterias

b2w/0.1

backdoorbot/1.0

barkrowler

beetlebot

black hole

blexbot

blowfish/1.0

blp_bbot

bluecoat

bookmark search tool

botalot

bubing

builtbottough

bullseye/1.0

bunnyslippers

butterfly

ccbot

cegbfeieh

centric

centricbot

cheesebot

cherrypicker

cherrypickerelite/1.0

cherrypickerse/1.0

cis455crawler

citeseerxbot

cliqzbot

compspybot

contextad

contextweb

copyrightcheck

cosmos

covario

covario-ids

crazywebcrawler-spider

crescent

cuil

custo

deusu

dibot+java

discobot

dittospyder

dlvr.it

docomo

dotbot

dropcatcher.de

educrawler

emailcollector

emailsiphon

emailwolf

erocrawler

esobisubscriber

exabot

extractorpro

ezooms

ezooms.bot

facebot

fairad client

femtosearchbot

findlinks

findlinks/2.1.5

flaming attackbot

foobot

fr_org_viewer

junkowtf.net
robots.txt