Log In Sign Up

cayhane.jp
robots.txt

Robots Exclusion Standard data for cayhane.jp

Resource Scan

Scan Details

Site Domain cayhane.jp
Base Domain cayhane.jp
Scan Status Ok
Last Scan2024-05-18T08:02:10+00:00
Next Scan 2024-06-17T08:02:10+00:00

Last Scan

Scanned2024-05-18T08:02:10+00:00
URL https://cayhane.jp/robots.txt
Redirect https://www.cayhane.jp/robots.txt
Redirect Domain www.cayhane.jp
Redirect Base cayhane.jp
Domain IPs 202.214.54.77
Redirect IPs 108.156.133.106, 108.156.133.18, 108.156.133.19, 108.156.133.26
Response IP 108.156.133.106
Found Yes
Hash 48848444ede9b8b9ae1e30a96140540b528a7582e7d826e796839912f5066222
SimHash 7b56dcb8ea31

Groups

acme.spider

Rule Path
Disallow /

agentname/0.1libwww-perl/6.02

Rule Path
Disallow /

alexa

Rule Path
Disallow /

alexandria prototype project

Rule Path
Disallow /

alexandria prototype+project

Rule Path
Disallow /

alexandria+prototype project

Rule Path
Disallow /

alexandria+prototype+project

Rule Path
Disallow /

allentrack

Rule Path
Disallow /

almaden

Rule Path
Disallow /

appie

Rule Path
Disallow /

arachmo

Rule Path
Disallow /

architext

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

arks

Rule Path
Disallow /

asterias

Rule Path
Disallow /

atomz

Rule Path
Disallow /

autoemailspider

Rule Path
Disallow /

awbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

bbot

Rule Path
Disallow /

bdfetch

Rule Path
Disallow /

biadu

Rule Path
Disallow /

biglotron

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3000

bjaaland

Rule Path
Disallow /

blaiz-bee

Rule Path
Disallow /

bloglines

Rule Path
Disallow /

blogpulse

Rule Path
Disallow /

boitho.com-dc

Rule Path
Disallow /

bookmark-manager

Rule Path
Disallow /

bot

Rule Path
Disallow /

-bot

Rule Path
Disallow /

brutus/aet

Rule Path
Disallow /

bspider

Rule Path
Disallow /

bwh3_user_agent

Rule Path
Disallow /

celestial

Rule Path
Disallow /

cfnetwork|checkbot

Rule Path
Disallow /

checkprivacy

Rule Path
Disallow /

chinaslocalsbrowses2.6

Rule Path
Disallow /

cloakdetect

Rule Path
Disallow /

codessampleswebsclient

Rule Path
Disallow /

combine

Rule Path
Disallow /

commons-httpclient

Rule Path
Disallow /

contentmatch

Rule Path
Disallow /

contentsmartz

Rule Path
Disallow /

core

Rule Path
Disallow /

coverscout

Rule Path
Disallow /

crawl

Rule Path
Disallow /

crawler

Rule Path
Disallow /

cursor

Rule Path
Disallow /

custo

Rule Path
Disallow /

datacha0s/2.0

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

demosbot

Rule Path
Disallow /

docomo

Rule Path
Disallow /

domain-crawler

Rule Path
Disallow /

download+master

Rule Path
Disallow /

dsurf

Rule Path
Disallow /

dtsearchspider

Rule Path
Disallow /

dumbot

Rule Path
Disallow /

easydl

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

exabot

Rule Path
Disallow /

fast-webcrawler

Rule Path
Disallow /

favorg

Rule Path
Disallow /

fdm 1

Rule Path
Disallow /

fdm+1

Rule Path
Disallow /

feedburner

Rule Path
Disallow /

feedfetcher

Rule Path
Disallow /

ferret

Rule Path
Disallow /

fetch api request

Rule Path
Disallow /

fetch api+request

Rule Path
Disallow /

fetch+api request

Rule Path
Disallow /

fetch+api+request

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

fulltext

Rule Path
Disallow /

funnelback

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

getright

Rule Path
Disallow /

geturl

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

girafabot

Rule Path
Disallow /

gnodspider

Rule Path
Disallow /

goldfire server

Rule Path
Disallow /

goldfire+server

Rule Path
Disallow /

grub

Rule Path
Disallow /

gulliver

Rule Path
Disallow /

harvest

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

hl_ftien_spider

Rule Path
Disallow /

holmes

Rule Path
Disallow /

htdig

Rule Path
Disallow /

htmlparser

Rule Path
Disallow /

httpcomponents/1.1

Rule Path
Disallow /

httpfetcher

Rule Path
Disallow /

httpget?5.2.2

Rule Path
Disallow /

httpget-5.2.2

Rule Path
Disallow /

httrack

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

iktomi

Rule Path
Disallow /

ilse

Rule Path
Disallow /

internetseer

Rule Path
Disallow /

intute

Rule Path
Disallow /

isilox

Rule Path
Disallow /

jakarta+commons-httpclient

Rule Path
Disallow /

java

Rule Path
Disallow /

jeeves

Rule Path
Disallow /

jobo

Rule Path
Disallow /

kyluka

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libcurl

Rule Path
Disallow /

libwww

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

lilina

Rule Path
Disallow /

linkbot

Rule Path
Disallow /

linkcheck

Rule Path
Disallow /

linkchecker

Rule Path
Disallow /

linklint-checkonly

Rule Path
Disallow /

linkscan

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

livejournal.com

Rule Path
Disallow /

lmspider

Rule Path
Disallow /

lockss

Rule Path
Disallow /

lwp

Rule Path
Disallow /

lwp::simple

Rule Path
Disallow /

lwp-request

Rule Path
Disallow /

lwp-tivial

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

lycos_

Rule Path
Disallow /

lycos+

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mail.ru_bot/fast/2.0

Rule Path
Disallow /

marcedit.5.2.web.client

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megite

Rule Path
Disallow /

milbot

Rule Path
Disallow /

mimas

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mnogosearch

Rule Path
Disallow /

moget

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

momspider

Rule Path
Disallow /

motor

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

muscatferre

Rule Path
Disallow /

myweb

Rule Path
Disallow /

nabot

Rule Path
Disallow /

nagios

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

netcraft

Rule Path
Disallow /

netluchs

Rule Path
Disallow /

ng/2.

Rule Path
Disallow /

ning

Rule Path
Disallow /

no_user_agent

Rule Path
Disallow /

nomad

Rule Path
Disallow /

nutch

Rule Path
Disallow /

ocelli

Rule Path
Disallow /

offline navigator

Rule Path
Disallow /

offline+navigator

Rule Path
Disallow /

onetszukaj

Rule Path
Disallow /

ourbrowser

Rule Path
Disallow /

parsijoo

Rule Path
Disallow /

pear.php.net

Rule Path
Disallow /

perman

Rule Path
Disallow /

php/

Rule Path
Disallow /

pioneer

Rule Path
Disallow /

playmusic.com

Rule Path
Disallow /

playstarmusic.com

Rule Path
Disallow /

powermarks

Rule Path
Disallow /

psbot

Rule Path
Disallow /

pycurl

Rule Path
Disallow /

python

Rule Path
Disallow /

qihoobot

Rule Path
Disallow /

rambler

Rule Path
Disallow /

readpaper

Rule Path
Disallow /

redalert|robozilla

Rule Path
Disallow /

repec.link.checker

Rule Path
Disallow /

robot

Rule Path
Disallow /

robots

Rule Path
Disallow /

rpt-httpclient/0.3-3e

Rule Path
Disallow /

rss

Rule Path
Disallow /

scan4mail

Rule Path
Disallow /

scientificcommons

Rule Path
Disallow /

scirus

Rule Path
Disallow /

scooter

Rule Path
Disallow /

seekbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

shoutcast

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

sogou

Rule Path
Disallow /

speedy

Rule Path
Disallow /

spider

Rule Path
Disallow /

spiderman

Rule Path
Disallow /

spiderview

Rule Path
Disallow /

steeler

Rule Path
Disallow /

strider

Rule Path
Disallow /

sunrise

Rule Path
Disallow /

superbot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

tailrank

Rule Path
Disallow /

technoratibot

Rule Path
Disallow /

teleport pro

Rule Path
Disallow /

teleport+pro

Rule Path
Disallow /

teoma

Rule Path
Disallow /

t-h-u-n-d-e-r-s-t-o-n-e

Rule Path
Disallow /

titan

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

ucsd

Rule Path
Disallow /

ultraseek

Rule Path
Disallow /

url2file

Rule Path
Disallow /

urlaliasbuilder

Rule Path
Disallow /

urllib

Rule Path
Disallow /

validator

Rule Path
Disallow /

virus_detector

Rule Path
Disallow /

virus+detector

Rule Path
Disallow /

voila

Rule Path
Disallow /

w3c-checklink

Rule Path
Disallow /

wanadoo

Rule Path
Disallow /

web downloader

Rule Path
Disallow /

web+downloader

Rule Path
Disallow /

webcloner

Rule Path
Disallow /

webcollage

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webcorpusbuilder

Rule Path
Disallow /

webinator

Rule Path
Disallow /

weblayers

Rule Path
Disallow /

webmetrics

Rule Path
Disallow /

webmirror

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wget

Rule Path
Disallow /

win16

Rule Path
Disallow /

wordpress

Rule Path
Disallow /

worm

Rule Path
Disallow /

www.gnip.com

Rule Path
Disallow /

www-mechanize

Rule Path
Disallow /

xenu

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

xenu link+sleuth

Rule Path
Disallow /

xenu+link sleuth

Rule Path
Disallow /

xenu+link+sleuth

Rule Path
Disallow /

y!j

Rule Path
Disallow /

yacy

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

zeus

Rule Path
Disallow /

zyborg

Rule Path
Disallow /
Back to top

Warnings

  • 14 invalid lines.
Back to top