unand.ac.id
robots.txt

Robots Exclusion Standard data for unand.ac.id

Resource Scan

Scan Details

Site Domain unand.ac.id
Base Domain unand.ac.id
Scan Status Ok
Last Scan2024-05-23T20:25:17+00:00
Next Scan 2024-06-22T20:25:17+00:00

Last Scan

Scanned2024-05-23T20:25:17+00:00
URL https://www.unand.ac.id/robots.txt
Domain IPs 103.151.89.156
Response IP 103.151.89.156
Found Yes
Hash 2cbef3686dd20c9f1d6ba8f3dbc2d592057dfac2985075a696d818eb92016569
SimHash a90558704579

Groups

xenu’s

Rule Path
Disallow /

xenu’s link sleuth 1.1c

Rule Path
Disallow /

httrack

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

*

Rule Path
Disallow /administrator/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Disallow /components/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /layouts/
Disallow /libraries/
Disallow /logs/
Disallow /modules/
Disallow /plugins/
Disallow /tmp/

mediapartners-google

Rule Path
Allow /

adsbot-google-mobile

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

googlebot

Rule Path
Allow /

feedfetcher-google

Rule Path
Allow /

ia_archiver

Rule Path
Allow /

scooter

Rule Path
Allow /

t-rex

Rule Path
Allow /

baiduspider

Rule Path
Allow /

bingbot

Rule Path
Allow /

huaweisymantecspider

Rule Path
Allow /

heise it-markt

Rule Path
Allow /

msnbot

Rule Path
Allow /

exabot

Rule Path
Allow /

msnbot-media

Rule Path
Allow /

nutch

Rule Path
Allow /

turnitinbot

Rule Path
Allow /

yahoo pipes 1.0

Rule Path
Allow /

yahoo! slurp

Rule Path
Allow /

yandexbot

Rule Path
Allow /

psbot

Rule Path
Allow /

ichiro

Rule Path
Allow /

yacy

Rule Path
Allow /

webauto

Rule Path
Allow /

webcrawler/3.0 robot libwww/5.0a

Rule Path
Allow /

webfetch

Rule Path
Allow /

voyager

Rule Path
Allow /

steeler

Rule Path
Allow /

snappy

Rule Path
Allow /

sensis web crawler

Rule Path
Allow /

mantraagent

Rule Path
Allow /

webcrawler

Rule Path
Allow /

fluffy the spider

Rule Path
Allow /

seosearch

Rule Path
Allow /

seoma

Rule Path
Allow /

seoma

Rule Path
Allow /

Comments

  • google
  • Alexa
  • altavista
  • baidu
  • Bing