useoul.edu
robots.txt

Robots Exclusion Standard data for useoul.edu

Resource Scan

Scan Details

Site Domain useoul.edu
Base Domain useoul.edu
Scan Status Ok
Last Scan2025-08-15T11:27:48+00:00
Next Scan 2025-09-14T11:27:48+00:00

Last Scan

Scanned2025-08-15T11:27:48+00:00
URL http://useoul.edu/robots.txt
Redirect https://en.snu.ac.kr/robots.txt
Redirect Domain en.snu.ac.kr
Redirect Base snu.ac.kr
Domain IPs 147.46.10.129
Redirect IPs 147.46.10.129
Response IP 147.46.10.129
Found Yes
Hash 47cc891d86175fed5908fb630ccc824c8f84fefd8d0c7dbfde3375fcf4aae802
SimHash d2165142670b

Groups

netestate

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

yandexcalendar

Rule Path
Disallow /

yandexmobilebot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

bomborabot

Rule Path
Disallow /

buck

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

paqlebot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

geedobot

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

criteobot/0.1

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

openindexspider

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

baiduspider
yisouspider
petalbot
bytespider
sogou web spider
sogou inst spider

Rule Path
Disallow /

proximic

Rule Path
Disallow /

ias_crawler

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

amazonbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

*

Rule Path
Allow /

Warnings

  • 1 invalid line.