Log In Sign Up

bone.jp
robots.txt

Robots Exclusion Standard data for bone.jp

Resource Scan

Scan Details

Site Domain bone.jp
Base Domain bone.jp
Scan Status Ok
Last Scan2024-09-17T19:48:49+00:00
Next Scan 2024-10-17T19:48:49+00:00

Last Scan

Scanned2024-09-17T19:48:49+00:00
URL https://bone.jp/robots.txt
Domain IPs 133.18.239.120
Response IP 133.18.239.120
Found Yes
Hash fa7f30c38196ce3cba50350e4ddd18b1b2154f0989279eafa35c508a99126ce8
SimHash 192693c91533

Groups

friendly_crawler

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

*

Rule Path
Disallow /p
Disallow /__
Disallow /dpc/12/
Disallow /dpc/dpc12
Disallow /dpc/icd12
Disallow /dpc/gogou12
Disallow /dpc/gogou-ope12
Disallow /dpc/14/
Disallow /dpc/dpc14
Disallow /dpc/icd14
Disallow /dpc/gogou14
Disallow /dpc/gogou-ope14
Disallow /dpc/16/
Disallow /dpc/dpc16
Disallow /dpc/icd16
Disallow /dpc/gogou16
Disallow /dpc/gogou-ope16
Disallow /dpc/dpc18
Disallow /dpc/18/
Disallow /dpc/icd18
Disallow /dpc/gogou18
Disallow /dpc/gogou-ope18
Disallow /dpc/19/
Disallow /dpc/dpc19
Disallow /dpc/icd19
Disallow /dpc/gogou19
Disallow /dpc/gogou-ope19
Disallow /dpc/20/
Disallow /dpc/dpc20
Disallow /dpc/icd20
Disallow /dpc/gogou20
Disallow /dpc/gogou-ope20
Disallow /dpc/22/
Disallow /dpc/dpc22
Disallow /dpc/icd22
Disallow /dpc/gogou22
Disallow /dpc/gogou-ope22

megalodon

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

adsbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /
Back to top

Comments

  • RFC 9309 Robots Exclusion Protocol
  • https://datatracker.ietf.org/doc/html/rfc9309#name-the-user-agent-line
  • https://www.rfc-editor.org/rfc/rfc9309.html
  • 240412 15:10 Friendly_Crawler, FriendlyCrawler [unknown runner]
  • General Rules (240422 move to top)
  • 221017 Disallow 2020 /dpc/20/*
  • 240706 Diallow 2022
  • Disallow Rules
  • Gyotaku
  • BLEXBot Crawler
  • 220220 09:09 https://dataforseo.com/dataforseo-bot
  • 220328 23:12 MJ12bot
  • 220328 23:15 AhrefsBot
  • 220404 SemrusBot From Cyprus 185.191.171.0/24
  • 220817 DotBot
  • 220831 Adsbot
  • 221123 serpstatbot
  • 230808 GPTBot (OpenAI / Microsoft)
  • 230818 Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)
  • 230826 Common Crawl https://commoncrawl.org/big-picture/frequently-asked-questions/
  • 230826 ChatGPT-User https://platform.openai.com/docs/plugins/bot
  • 230918 Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot
  • 231101 NICT ICC-Crawler https://ucri.nict.go.jp/icccrawler.html
  • 240329 Seekport Bot https://bot.seekport.com/
  • 240412 15:10 Friendly_Crawler, FriendlyCrawler [unknown runner]
  • User-agent: Friendly_Crawler
  • Disallow: /
  • User-agent: FriendlyCrawler
  • Disallow: /
  • 240414 19:52
  • User-agent: Friendly_Crawler/2.0
  • Disallow: /
  • 240415 06:30 remark 240422 21:20
  • User-agent: friendlycrawler
  • Disallow: /
  • Apache Nutch https://nutch.apache.org/community/bot/
  • 240415 11:56 remark 240423 10:57
  • reactivate 240427 21:25
  • 240419 CloudeBot
  • 240420
  • 240419 google Gemini App and Vertex AI
  • 240420
  • 240420
  • 240420
  • 220328 23:15 AhrefsBot
  • User-agent: AhrefsBot
  • Disallow: /
  • 240423 Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io)
  • via bc.googleusercontent.com. (proxy)
Back to top