Log In Sign Up

june.so
robots.txt

Robots Exclusion Standard data for june.so

Resource Scan

Scan Details

Site Domain june.so
Base Domain june.so
Scan Status Ok
Last Scan2024-09-06T19:41:14+00:00
Next Scan 2024-10-06T19:41:14+00:00

Last Scan

Scanned2024-09-06T19:41:14+00:00
URL https://june.so/robots.txt
Redirect https://www.june.so/robots.txt
Redirect Domain www.june.so
Redirect Base june.so
Domain IPs 76.76.21.21
Redirect IPs 76.76.21.241, 76.76.21.9
Response IP 76.76.21.9
Found Yes
Hash 5718610bd33d623b50ec31e552c3eb3248cacc4c6df901b5a38ad0d536cad310
SimHash 4355520b47a6

Groups

*

Rule Path
Allow /

*

Rule Path
Disallow /coupon/

ruby

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

aqua_products

Rule Path
Disallow /

asterias

Rule Path
Disallow /

b2w

Rule Path
Disallow /

backdoorbot

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

bloglovin

Rule Path
Disallow /

blowfish

Rule Path
Disallow /

bookmark search tool

Rule Path
Disallow /

bot

Rule Path
Disallow /

botalot

Rule Path
Disallow /

builtbottough

Rule Path
Disallow /

bullseye

Rule Path
Disallow /

bunnyslippers

Rule Path
Disallow /

calculon spider

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

cherrypickerelite

Rule Path
Disallow /

cherrypickerse

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

copernic

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

cosmos

Rule Path
Disallow /

crescent

Rule Path
Disallow /

daum

Rule Path
Disallow /

dittospyder

Rule Path
Disallow /

dumbot

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

enterprise_search

Rule Path
Disallow /

erocrawler

Rule Path
Disallow /

es

Rule Path
Disallow /

exabot

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

fairad client

Rule Path
Disallow /

fatbot

Rule Path
Disallow /

flaming attackbot

Rule Path
Disallow /

foobot

Rule Path
Disallow /

freefind

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

getright

Rule Path
Disallow /

grub

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

harvest

Rule Path
Disallow /

hatena antenna

Rule Path
Disallow /

hloader

Rule Path
Disallow /

httplib

Rule Path
Disallow /

humanlinks

Rule Path
Disallow /

idg

Rule Path
Disallow /

infonavirobot

Rule Path
Disallow /

inoreader.com

Rule Path
Disallow /

iron33

Rule Path
Disallow /

jennybot

Rule Path
Disallow /

jetbot

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

kenjin spider

Rule Path
Disallow /

keyword density

Rule Path
Disallow /

larbin

Rule Path
Disallow /

lexibot

Rule Path
Disallow /

libweb/clshttp

Rule Path
Disallow /

linkextractorpro

Rule Path
Disallow /

linkscan

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

lnspiderguy

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

mata hari

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

microsoft url control

Rule Path
Disallow /

miixpc

Rule Path
Disallow /

mister pix

Rule Path
Disallow /

moget

Rule Path
Disallow /

spbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

naver

Rule Path
Disallow /

netants

Rule Path
Disallow /

netmechanic

Rule Path
Disallow /

nicerspro

Rule Path
Disallow /

nutch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

omniexplorer_bot

Rule Path
Disallow /

openbot

Rule Path
Disallow /

openfind

Rule Path
Disallow /

openfind data gathere

Rule Path
Disallow /

optimizer

Rule Path
Disallow /

oracle ultra search

Rule Path
Disallow /

parser

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

pcore-http

Rule Path
Disallow /

perman

Rule Path
Disallow /

php

Rule Path
Disallow /

propowerbot

Rule Path
Disallow /

prowebwalker

Rule Path
Disallow /

psbot

Rule Path
Disallow /

pu_in crawler

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

pyton-requests

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

queryn metasearch

Rule Path
Disallow /

radiation retriever

Rule Path
Disallow /

repomonkey

Rule Path
Disallow /

repomonkey bait & tackle

Rule Path
Disallow /

rma

Rule Path
Disallow /

searchpreview

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sootle

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

spankbot

Rule Path
Disallow /

spanner

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

stanford

Rule Path
Disallow /

stanford comp sci

Rule Path
Disallow /

suzuran

Rule Path
Disallow /

swiftbot

Rule Path
Disallow /

szukacz

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

telesoft

Rule Path
Disallow /

the intraformant

Rule Path
Disallow /

thenomad

Rule Path
Disallow /

tocrawl

Rule Path
Disallow /

trident

Rule Path
Disallow /

true_robot

Rule Path
Disallow /

turingos

Rule Path
Disallow /

updownerbot

Rule Path
Disallow /

url control

Rule Path
Disallow /

url_spider_pro

Rule Path
Disallow /

urly warning

Rule Path
Disallow /

vci

Rule Path
Disallow /

vci webviewer vci webviewer win32

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

web image collector

Rule Path
Disallow /

webauto

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webenhancer

Rule Path
Disallow /

websauger

Rule Path
Disallow /

website quester

Rule Path
Disallow /

webster pro

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webvac

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wget

Rule Path
Disallow /

willybot

Rule Path
Disallow /

woorankreview

Rule Path
Disallow /

www-collector-e

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

yak

Rule Path
Disallow /

zeus

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

news-please

Rule Path
Disallow /

perplexity-ai

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.june.so/sitemap.xml
Back to top

Comments

  • *
  • *
  • Ruby
  • python-requests
  • libwww-perl
  • DotBot
  • Mail.RU_Bot
  • +https://www.megaindex.ru/
  • Alexibot
  • Aqua_Products
  • asterias
  • b2w
  • BackDoorBot
  • BecomeBot
  • Bloglovin
  • BlowFish
  • Bookmark search tool
  • Bot
  • BotALot
  • BuiltBotTough
  • Bullseye
  • BunnySlippers
  • calculon spider
  • CheeseBot
  • CherryPicker
  • CherryPickerElite
  • CherryPickerSE
  • coccoc
  • Copernic
  • CopyRightCheck
  • cosmos
  • Crescent
  • Daum
  • DittoSpyder
  • dumbot
  • EmailCollector
  • EmailSiphon
  • EmailWolf
  • Enterprise_Search
  • EroCrawler
  • es
  • exabot
  • ExtractorPro
  • ezooms
  • FairAd Client
  • fatbot
  • Flaming AttackBot
  • Foobot
  • FreeFind
  • Gaisbot
  • GetRight
  • grub
  • grub-client
  • Harvest
  • Hatena Antenna
  • hloader
  • httplib
  • humanlinks
  • IDG
  • InfoNaviRobot
  • inoreader.com
  • Iron33
  • JennyBot
  • Jetbot
  • jikespider
  • Kenjin Spider
  • Keyword Density
  • larbin
  • LexiBot
  • libWeb/clsHTTP
  • LinkextractorPro
  • LinkScan
  • LinkWalker
  • LNSpiderguy
  • lwp-trivial
  • Mata Hari
  • MegaIndex.ru
  • Microsoft URL Control
  • MIIxpc
  • Mister PiX
  • moget
  • spbot
  • MSIECrawler
  • naver
  • NetAnts
  • NetMechanic
  • NICErsPRO
  • Nutch
  • Offline Explorer
  • OmniExplorer_Bot
  • Openbot
  • Openfind
  • Openfind data gathere
  • Optimizer
  • Oracle Ultra Search
  • Parser
  • PaperLiBot
  • Pcore-HTTP
  • PerMan
  • PHP
  • ProPowerBot
  • ProWebWalker
  • psbot
  • Pu_iN Crawler
  • Python-urllib
  • pyton-requests
  • Qwantify
  • QueryN Metasearch
  • Radiation Retriever
  • RepoMonkey
  • RepoMonkey Bait & Tackle
  • RMA
  • searchpreview
  • SeznamBot
  • sitebot
  • SiteSnagger
  • SMTBot
  • Sogou
  • sootle
  • sosospider
  • SpankBot
  • spanner
  • Spinn3r
  • Stanford
  • Stanford Comp Sci
  • suzuran
  • Swiftbot
  • Szukacz
  • Teleport
  • TeleportPro
  • Telesoft
  • The Intraformant
  • TheNomad
  • toCrawl
  • Trident
  • True_Robot
  • turingos
  • Updownerbot
  • URL Control
  • URL_Spider_Pro
  • URLy Warning
  • VCI
  • VCI WebViewer VCI WebViewer Win32
  • voilabot
  • Web Image Collector
  • WebAuto
  • WebBandit
  • WebCopier
  • WebEnhancer
  • WebSauger
  • Website Quester
  • Webster Pro
  • WebStripper
  • WebVac
  • WebZip
  • Wget
  • willybot
  • woorankreview
  • WWW-Collector-E
  • yodaobot
  • YaK
  • Zeus
  • ichiro
  • NaverBot
  • Yeti
  • ChatGPT-User
  • GPTBot
  • CCBot
  • anthropic-ai
  • cohere-ai
  • ia_archiver
  • omgili
  • omgilibot
  • MJ12bot
  • PiplBot
  • Bytespider
  • ClaudeBot
  • Claude-Web
  • FacebookBot
  • magpie-crawler
  • NewsNow
  • news-please
  • Perplexity-ai
  • Host
  • Sitemaps
Back to top

Warnings

  • 2 invalid lines.
  • `host` is not a known field.
Back to top