bokete.jp
robots.txt

Robots Exclusion Standard data for bokete.jp

Resource Scan

Scan Details

Site Domain bokete.jp
Base Domain bokete.jp
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-06T22:03:13+00:00
Next Scan 2024-12-05T22:03:13+00:00

Last Successful Scan

Scanned2024-05-10T21:51:08+00:00
URL https://bokete.jp/robots.txt
Domain IPs 18.155.68.103, 18.155.68.117, 18.155.68.24, 18.155.68.36, 2600:9000:23d2:2600:9:8d41:bc0:93a1, 2600:9000:23d2:7400:9:8d41:bc0:93a1, 2600:9000:23d2:a800:9:8d41:bc0:93a1, 2600:9000:23d2:b200:9:8d41:bc0:93a1, 2600:9000:23d2:bc00:9:8d41:bc0:93a1, 2600:9000:23d2:e200:9:8d41:bc0:93a1, 2600:9000:23d2:f600:9:8d41:bc0:93a1, 2600:9000:23d2:f800:9:8d41:bc0:93a1
Response IP 18.155.68.103
Found Yes
Hash d676c69abc86a642781924b79fef9b3458ea1b31e11cf57c6d81025e130b39ce
SimHash 50174060e7bb

Groups

*

Rule Path
Disallow

twitterbot

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

mappy

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

feedwordpress

Rule Path
Disallow /

yandexantivirus

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yeti

Rule Path
Disallow /

proximic

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

ia_archiver

Rule Path
Disallow /