hku.edu.tr
robots.txt

Robots Exclusion Standard data for hku.edu.tr

Resource Scan

Scan Details

Site Domain hku.edu.tr
Base Domain hku.edu.tr
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-19T00:06:06+00:00
Next Scan 2024-12-18T00:06:06+00:00

Last Successful Scan

Scanned2023-02-04T09:00:00+00:00
URL https://hku.edu.tr/robots.txt
Redirect https://www.hku.edu.tr/robots.txt
Redirect Domain www.hku.edu.tr
Redirect Base hku.edu.tr
Domain IPs 141.193.213.10, 141.193.213.11
Redirect IPs 141.193.213.10, 141.193.213.11
Response IP 141.193.213.11
Found Yes
Hash 0bb9ad46a47912d796c2328f1adad479b8e8343505e6dfcf97ad793f40be8b9d
SimHash 5240090ecbba

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

yandex

Rule Path
Disallow /

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

synapse

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

cb/nutch

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

Comments

  • Goo Japonya
  • Naver Kore
  • Baidu Çin
  • SoGou Çin
  • Youdao Çin
  • EasouSpider Çin
  • Synapse Ukrayna
  • MJ12bot UK
  • XoviBot Almanya
  • Mail.RU_Botu engelleme
  • SISTRIX seo aracı botu
  • cb/nutch
  • Python-urllib #Celilcan tarafından oluşturuldu.
  • CCBot Apache Bot ve Crawler tarama
  • AhrefsBot engelleme