kbbi.kemdikbud.go.id
robots.txt

Robots Exclusion Standard data for kbbi.kemdikbud.go.id

Resource Scan

Scan Details

Site Domain kbbi.kemdikbud.go.id
Base Domain kemdikbud.go.id
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-02-23T08:02:49+00:00
Next Scan 2025-03-25T08:02:49+00:00

Last Successful Scan

Scanned2025-01-02T08:01:53+00:00
URL https://kbbi.kemdikbud.go.id/robots.txt
Domain IPs 104.18.24.30, 104.18.25.30, 2606:4700::6812:181e, 2606:4700::6812:191e
Response IP 104.18.25.30
Found Yes
Hash 43d66877160055b19678c078bc1d74b8cfeaed1dae16ab6ef498828b2e226f36
SimHash d9905e14519f

Groups

python-requests

Rule Path
Disallow /entri

Other Records

Field Value
crawl-delay 100000

bot*

Rule Path
Disallow /entri

Other Records

Field Value
crawl-delay 100000

bingbot

Rule Path
Disallow /entri

coibotparser/2.0

Rule Path
Disallow /entri

twitterbot/1.0

Rule Path
Disallow /entri

bingbot/2.0

Rule Path
Disallow /entri

ahrefsbot/7.0

Rule Path
Disallow /entri

semrushbot/7~bl

Rule Path
Disallow /entri

yandexbot/3.0

Rule Path
Disallow /entri

semanticscholarbot

Rule Path
Disallow /entri

discordbot/2.0

Rule Path
Disallow /entri

dotbot/1.2

Rule Path
Disallow /entri

paperlibot/2.1

Rule Path
Disallow /entri

archive.org_bot

Rule Path
Disallow /entri

telegrambot

Rule Path
Disallow /entri

trokambot/1.0

Rule Path
Disallow /entri

serendeputybot/0.8.6

Rule Path
Disallow /entri

mediumbot-metatagfetcher/0.3

Rule Path
Disallow /entri

scribdbot-pdflinkchecker/1.0

Rule Path
Disallow /entri

slackbot

Rule Path
Disallow /entri

facebot

Rule Path
Disallow /entri

mediumbot-metatagfetcher/0.3

Rule Path
Disallow /entri

siteauditbot/0.97

Rule Path
Disallow /entri

sitescorebot

Rule Path
Disallow /entri

mojeekbot/0.11

Rule Path
Disallow /entri

gptbot

Rule Path
Disallow /entri

dataforseobot/1.0

Rule Path
Disallow /entri

imagesiftbot

Rule Path
Disallow /entri

Warnings

  • 6 invalid lines.