Log In Sign Up

on-o.com
robots.txt

Robots Exclusion Standard data for on-o.com

Resource Scan

Scan Details

Site Domain on-o.com
Base Domain on-o.com
Scan Status Ok
Last Scan2024-09-21T07:30:22+00:00
Next Scan 2024-09-28T07:30:22+00:00

Last Scan

Scanned2024-09-21T07:30:22+00:00
URL https://on-o.com/robots.txt
Domain IPs 118.240.80.64, 240d:1a:17c:3a00:c4b6:f012:c59:b72c
Response IP 118.240.80.64
Found Yes
Hash 7796b219fa0656b4a3b5e61bd8d43bdaf1ed802dc56826cae62f172abbeaa2a9
SimHash 416a7df0df0b

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /mailman/
Disallow /pipermail/
Disallow /mail/
Disallow /webmail/
Disallow /webmin/
Disallow /munin/
Disallow /capture/

bingbot
moget

Rule Path
Disallow /cgi-bin/

googlebot

Rule Path
Disallow /cgi-bin/
Disallow /scuttle/bookmarks/
Disallow /scuttle/populartags/
Disallow /scuttle/tags/
Disallow /scuttle/tags.php/

hatena antenna

Rule Path
Disallow /cgi-bin/

slurp

Rule Path
Disallow /cgi-bin/

slurp

Rule Path
Disallow /

pockey-gethtml

Rule Path
Disallow /

rufusbot

Rule Path
Disallow /page/diary/

baiduspider+

Rule Path
Disallow /

yeti

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 300

exabot

Rule Path
Disallow /

tversity media server

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

linguee

Rule Path
Disallow /scuttle/

ahrefsbot

Rule Path
Disallow /scuttle/

blexbot

Rule Path
Disallow /scuttle/

petalbot

Rule Path
Disallow /scuttle/

yandexbot

Rule Path
Disallow /scuttle/

petalbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /
Back to top

Comments

  • MSN msnbot
  • User-agent: searchpreview
  • Disallow: /
  • User-Agent: Baiduspider
  • Disallow: /
  • http://help.naver.com/robots/
  • ttp://www.exabot.com/go/robot
  • ec2-23-20-240-200.compute-1.amazonaws.com - - [30/Mar/2012:16:00:59 +0900] "GET /~tkyn/wiki/?plugin=attach&refer=JDeveloper%2010g%2FHello%20World&openfile=NewClass3.png HTTP/1.1" 200 27297 "" "EC2LinkFinder" 23.20.240.200
  • User-Agent: EC2LinkFinder
  • Crawl-Delay: 30
  • s0106306023d6415f.ed.shawcable.net - - [14/Oct/2014:12:27:24 +0900] "GET /FirstWeblog/archives/000253.html HTTP/1.1" 403 918 "-" "TVersity Media Server 2.4 Indexer" 174.3.137.153
  • 2018/01/30
  • 2018/12/03 Too Many Scuttle Access to MySQL heaby
  • https://www.linguee.com/bot
  • 2018/12/07 https://ahrefs.com/robot
  • 2019/01/15
  • 2020/08/22
  • 2021/12/23 yandex.ru
  • 2023/3/5
  • 2023/10/10
Back to top

Warnings

  • `clawl-delay` is not a known field.
Back to top