Log In Sign Up

headjapan.com
robots.txt

Robots Exclusion Standard data for headjapan.com

Resource Scan

Scan Details

Site Domain headjapan.com
Base Domain headjapan.com
Scan Status Ok
Last Scan2024-11-03T06:46:46+00:00
Next Scan 2024-12-03T06:46:46+00:00

Last Scan

Scanned2024-11-03T06:46:46+00:00
URL https://headjapan.com/robots.txt
Domain IPs 160.16.91.60
Response IP 160.16.91.60
Found Yes
Hash 474250c07d7fc2870c591f72bffa310e28d710761efcb3a48d611b687e8f22d8
SimHash 4397f7835d67

Groups

a6-indexer

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

a6-indexer/1.0

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

acoon

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

aghaven

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

aghaven/nutch-1.2

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

alexibot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

appcodescrawler

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

applewebkit/412

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

archive.org_bot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

aqua_products

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

asterias

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

b2w/0.1

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

backdoorbot/1.0

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

baidu

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

beetlebot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

blexbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

black hole

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

blowfish/1.0

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

bluecoat

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

bookmark search tool

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

botalot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

builtbottough

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

bullseye/1.0

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

bunnyslippers

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

butterfly

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

ccbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

cegbfeieh

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

centric

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

centricbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

cheesebot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

cherrypicker

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

cherrypickerelite/1.0

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

cherrypickerse/1.0

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

cis455crawler

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

citeseerxbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

coccoc

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

compspybot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

contextad

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

contextweb

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

copyrightcheck

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

cosmos

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

covario

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

covario-ids

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

crescent

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

cuil

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

custo

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

dibot+java

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

discobot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

dittospyder

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

dotbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

educrawler

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

emailcollector

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

emailsiphon

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

emailwolf

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

erocrawler

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

esobisubscriber

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

exabot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

extractorpro

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

ezooms

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

ezooms.bot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

facebot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

fairad client

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

findlinks

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

findlinks/2.1.5

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

flaming attackbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

foobot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

fr_org_viewer

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

gaisbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

garlik

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

garlikcrawler

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

getright/4.2

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

gingercrawler

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

grabber

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

grub

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

grub-client

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

harvest/1.5

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

hloader

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

httplib

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

humanlinks

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

ia_archiver

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

ichiro

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

ichiro/mobile

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

ichiro/mobile goo

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

infonavirobot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

infopath

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

inktomi

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

inktomisearch

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

www.integromedb.org/crawler

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

iron33/1.0.2

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

itah

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

jennybot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

jikespider

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

kenjin spider

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

keyword density/0.9

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

larbin

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

lemurwebcrawler

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

lexibot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

libweb/clshttp

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

linkcheck

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

linkextractorpro

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

linkscan

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

linkwalker

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

lnspiderguy

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

lssbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

lwp-trivial

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

mail.ru

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

mail.ru_bot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

mata hari

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

mediapartners-google

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

megaupload

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

merchantcentric

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

merchantcentricbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

microsoft url control

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

miixpc

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

minerva

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

mister pix

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

moget

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

mozilla

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

msiecrawler

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

msnbot-media

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

naverbot/1.0

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

netants

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

netmechanic

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

nicerspro

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

nutch

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

nutraspace

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

obot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

ocelli

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

offline explorer

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

openbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

openfind

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

oracle ultra search

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

panscient.com

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

perman

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

picsearch

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

propowerbot/2.14

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

prowebwalker

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

psbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

ptd-crawler

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

python-urllib

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

queryn metasearch

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

radiation retriever 1.1

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

replazbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

repomonkey

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

rma

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

ru_bot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

scooperbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

scoutjet

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

screaming frog seo spider

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

searchmetricsbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

searchpreview

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

semrush

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

seznambot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

shopwiki

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

sistrix

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

sitebot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

sitecheck

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

siteimprove

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

sitesnagger

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

slurp

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

smtbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

soa

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

soabot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

soasearch

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

sobisubscriber

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

sogou

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

sogou spider

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

solomono

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

solomonobot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

sosospider

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

spankbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

spanner

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

spbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

speedy

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

sputnikbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

suzuran

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

szukacz/1.4

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

teleport

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

teleportpro

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

telesoft

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

teoma

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

the intraformant

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

thenomad

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

tighttwatbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

titan

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

tocrawl/urldispatcher

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

trident/4.0

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

true_robot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

turingos

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

twiceler

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

url control

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

urldispatcher

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

url_spider_pro

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

urly warning

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

vci

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

visbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

visvo

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

voilabot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

web image collector

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

webauto

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

webbandit

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

webcopier

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

webenhancer

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

webmasterworldforumbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

websauger

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

website quester

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

webster pro

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

webstripper

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

webvac

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

webzip

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

wesee

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

wget

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

wotbox

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

www-collector-e

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

xovibot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

yahooseeker

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

yandex

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

yanga+worldsearch+bot+v1.1

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

yeti

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

yodaobot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

youdaobot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

yyspider

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

zeus

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

zibb

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

zibber

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

zumbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

megalodon

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10
Back to top

Warnings

  • 3 invalid lines.
Back to top