retro-girl.jp
robots.txt
Robots Exclusion Standard data for retro-girl.jp
Resource Scan
Scan Details
| Site Domain | retro-girl.jp |
| Base Domain | retro-girl.jp |
| Scan Status | Ok |
| Last Scan | 2024-10-26T07:55:26+00:00 |
| Next Scan | 2024-11-25T07:55:26+00:00 |
Last Scan
| Scanned | 2024-10-26T07:55:26+00:00 |
| URL | https://www.retro-girl.jp/robots.txt |
| Domain IPs | 151.101.131.52, 151.101.195.52, 151.101.3.52, 151.101.67.52 |
| Response IP | 199.232.47.52 |
| Found | Yes |
| Hash | 32cab0dfd247ef56b195d8109d9c76263ca7e40a13a47327eadc87a2eb15a1fb |
| SimHash | 591c75d28483 |
Groups
a6-indexer
aboundex
aboundexbot
adsbot
advbot
ahrefsbot
aihitbot
alphabot
alphaseobot
alphaseobot-sa
amazonbot
applebot
archive.org_bot
awariobot
awariorssbot
awariosmartbot
baiduspider
baiduspider-image
baiduspider-video
barkrowler
bbot
betabot
bingbot
blexbot
brands-bot-logo
bubing
bytespider
ccbot
chatgpt-user
checkmarknetwork
cincraw
clarabot
clark-crawler
claude
claudebot
cliqzbot
coccocbot
coccocbot-image
coccocbot-web
contacts-crawler
crawler4j
dataforseobot
datanyze
dataprovider
daum
diffbot
dnyzbot
domaincrawler
domainstatsbot
dotbot
easouspider
ec2linkfinder
etaospider
exabot
experibot
extlinksbot
ezooms
ezooms robot
femtosearchbot
fidget-spinner-bot
findxbot
flfbaldrbot
funnelback
garlikcrawler
geedobot
gocrawl
go-http-client
gptbot
gsa-crawler
haosouspider
hubspot
ia_archiver
icc-crawler
idbot
iframely
iframely/1.3.1
indeedbot
inetdex-bot
infopath
infopath.2
infotiger
infotigerbot
ioncrawl
ip-web-crawler.com
jambot
jamesbot
jikespider
jobboersebot
keybot translation-search-machine
linespider
linguee
linkpadbot
lipperhey
livelapbot
loadtimebot
lssrocketcrawler
ltx71
macocu
magpie-crawler
mail.ru
mappy
mauibot
meanpathbot
megaindex
megaindex.ru
megaindex.ru/2.0
memorybot
mixrankbot
mj12bot
mojeekbot
msnbot
naverbot
neevabot
netestate ne crawler
netestate ne crawler (+http://www.website-datenbank.de/)
nextgensearchbot
niocbot
nutch
obot
okhttp
owler
panscient.com
parsijoo-bot
perl lwp
petalbot
pimonster
piplbot
plukkie
psbot
qwantify
rankactivelinkbot
ranksonicbot
riddler
rogerbot
rukicrawler
safednsbot
scrapy
searchmetericsbot
seekbot
seeker
seekport
seekportbot
semanticscholarbot
semrushbot
senutobot
seokicks
seokicks-robot
serpstatbot
serptimizerbot
seznambot
sidetrade indexer bot
sistrix
sistrix crawler
sitebot
siteexplorer
slack-imgproxy
slurp
smtbot
sogou blog
sogou inst spider
sogou news spider
sogou orion spider
sogou spider2
sogou web spider
sosospider
spbot
stractbot
stractbot/0.1
surdotlybot
swebot
taptubot
tbot-nutch
the knowledge ai
thunderstone
tinytestbot
turnitin robot
turnitinbot
twengabot
twiceler
twitterbot
uptimebot
uptimerobot/2.0
velenpublicwebcrawler
vscooter
vuhuvbot
wbsearchbot
webprosbot
wiseguys robot
woorankreview
worldwebheritage.org
worldwebheritage.org/1.0
wotbox
xovibot
yacybot
yak
yandex
yeti
yeti/1.1
yisouspider
yoozbot
youdaobot
zoominfobot
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 120 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.retro-girl.jp/sitemap.xml |
Warnings
- 5 invalid lines.