Log In Sign Up

repository.kulib.kyoto-u.ac.jp
robots.txt

Robots Exclusion Standard data for repository.kulib.kyoto-u.ac.jp

Resource Scan

Scan Details

Site Domain repository.kulib.kyoto-u.ac.jp
Base Domain kyoto-u.ac.jp
Scan Status Ok
Last Scan2024-11-03T12:13:55+00:00
Next Scan 2024-12-03T12:13:55+00:00

Last Scan

Scanned2024-11-03T12:13:55+00:00
URL https://repository.kulib.kyoto-u.ac.jp/robots.txt
Domain IPs 133.3.250.113
Response IP 133.3.250.113
Found Yes
Hash d08b39ddc8eb7fd08074ab45ee5cf89e8952cc65e51b9fdb93843d7a1c9c33c6
SimHash 2414c9244aeb

Groups

*

Rule Path
Disallow /dspace/browse-subject*
Disallow /dspace/browse-author*
Disallow /dspace/items-by-author*
Disallow /dspace/items-by-subject*
Disallow /dspace/password-login

baiduspider

Rule Path
Disallow /

baiduimagespider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

ichiro

Rule Path
Disallow /
Back to top

Comments

  • Disallow: /dspace/browse-*
  • User-agent: Googlebot
  • Disallow: /dspace/items-by-author*
  • Disallow: /dspace/items-by-subject*
  • Disallow: /dspace/password-login
  • baiduクローラーが回ってこないようにする(2012.10.1).
  • 参照：http://www.oratorio-tangram.com/WebSec/baidu403.html
  • User-agent: bingbot
  • Crawl-delay: 1800
  • User-agent: Googlebot
  • Crawl-delay: 7200
  • User-agent: Googlebot-Mobile
  • Crawl-delay: 7200
  • bingbotとともにmsnbotも排除(2012.10.2）.
  • User-agent: msnbot
  • Crawl-daily: 1800
  • "Sogou web spider"を排除（2012.10.2）.
  • "Yandex"（ロシアの検索エンジン）を排除（2012.10.5）.
  • "ichiro"（gooの検索エンジン）を排除（2012.10.5）.
Back to top