kaigodb.com
robots.txt

Robots Exclusion Standard data for kaigodb.com

Resource Scan

Scan Details

Site Domain kaigodb.com
Base Domain kaigodb.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-20T10:36:48+00:00
Next Scan 2024-11-19T10:36:48+00:00

Last Successful Scan

Scanned2024-08-29T10:35:28+00:00
URL https://kaigodb.com/robots.txt
Domain IPs 45.60.67.93
Response IP 45.60.67.93
Found Yes
Hash c0c9a8f9d5f3a63a0f26afcfefb3088f53a6702de39326825c0ca8ee975936d5
SimHash 613094fae3e3

Groups

*

Rule Path
Disallow /favorite_jigyousho_and_services/add
Disallow /favorite_jigyousho_and_services/add*
Disallow /ga.php
Disallow /ga.php*
Disallow /redirect_for_ae/
Disallow /redirect_for_ae/*
Disallow /jigyousho/1373001864-430/
Disallow /jigyousho/1170900318-720/
Disallow /jigyousho/1170900318-740/
Disallow /search/

ahrefsbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

crawler

Rule Path
Disallow /

outfoxbot

Rule Path
Disallow /

accelatech rsscrawler

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

blogshot

Rule Path
Disallow /

aipbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

irlbot

Rule Path
Disallow /

just-crawler

Rule Path
Disallow /

rewriterule

Rule Path
Disallow /

steeler

Rule Path
Disallow /

wget

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

hatena

Rule Path
Disallow /

hatena antenna

Rule Path
Disallow /

lwp

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

naver

Rule Path
Disallow /

nhnbot

Rule Path
Disallow /

naverbot-1.0

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

voyager

Rule Path
Disallow /

w3c-webcon

Rule Path
Disallow /

website explorer

Rule Path
Disallow /

webox

Rule Path
Disallow /

webauto

Rule Path
Disallow /

webfetch

Rule Path
Disallow /

yandex

Rule Path
Disallow /

psbot

Rule Path
Disallow /

pockey-gethtml

Rule Path
Disallow /

rss_reader

Rule Path
Disallow /

sakebot

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

veoh

Rule Path
Disallow /

webalta

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://kaigodb.com/site_map/sitemap.xml