sp.bokete.jp
robots.txt

Robots Exclusion Standard data for sp.bokete.jp

Resource Scan

Scan Details

Site Domain sp.bokete.jp
Base Domain bokete.jp
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-25T12:40:07+00:00
Next Scan 2024-06-08T12:40:07+00:00

Last Successful Scan

Scanned2024-05-10T12:15:53+00:00
URL https://sp.bokete.jp/robots.txt
Redirect https://bokete.jp/robots.txt
Redirect Domain bokete.jp
Redirect Base bokete.jp
Domain IPs 13.225.4.129, 13.225.4.47, 13.225.4.54, 13.225.4.89, 2600:9000:21b4:2000:12:e59c:f780:93a1, 2600:9000:21b4:2800:12:e59c:f780:93a1, 2600:9000:21b4:6000:12:e59c:f780:93a1, 2600:9000:21b4:6a00:12:e59c:f780:93a1, 2600:9000:21b4:6c00:12:e59c:f780:93a1, 2600:9000:21b4:aa00:12:e59c:f780:93a1, 2600:9000:21b4:ac00:12:e59c:f780:93a1, 2600:9000:21b4:ee00:12:e59c:f780:93a1
Redirect IPs 108.157.254.120, 108.157.254.41, 108.157.254.47, 108.157.254.67, 2600:9000:2753:2600:9:8d41:bc0:93a1, 2600:9000:2753:2e00:9:8d41:bc0:93a1, 2600:9000:2753:5400:9:8d41:bc0:93a1, 2600:9000:2753:6400:9:8d41:bc0:93a1, 2600:9000:2753:6a00:9:8d41:bc0:93a1, 2600:9000:2753:ca00:9:8d41:bc0:93a1, 2600:9000:2753:de00:9:8d41:bc0:93a1, 2600:9000:2753:f800:9:8d41:bc0:93a1
Response IP 108.157.254.47
Found Yes
Hash d676c69abc86a642781924b79fef9b3458ea1b31e11cf57c6d81025e130b39ce
SimHash 50174060e7bb

Groups

*

Rule Path
Disallow

twitterbot

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

mappy

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

feedwordpress

Rule Path
Disallow /

yandexantivirus

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yeti

Rule Path
Disallow /

proximic

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

ia_archiver

Rule Path
Disallow /