www1.sun.ac.za
robots.txt

Robots Exclusion Standard data for www1.sun.ac.za

Resource Scan

Scan Details

Site Domain www1.sun.ac.za
Base Domain sun.ac.za
Scan Status Ok
Last Scan2025-03-19T23:37:13+00:00
Next Scan 2025-04-18T23:37:13+00:00

Last Scan

Scanned2025-03-19T23:37:13+00:00
URL https://www1.sun.ac.za/robots.txt
Domain IPs 146.232.66.101
Response IP 146.232.66.101
Found Yes
Hash 3f72ee1ee7abdf65b54c7ad9708d7521d560a8de8c936e010f70af1ec78159d1
SimHash 234347287355

Groups

linkedinbot/1.0

Rule Path
Disallow /

amazonbot/0.1

Rule Path
Disallow /

meta-externalagent/1.1

Rule Path
Disallow /

ccbot/2.0

Rule Path
Disallow /

twitterbot/1.0

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

adsbot-google-mobile

Rule Path
Disallow /

chatgpt-user/1.0

Rule Path
Disallow /

googlebot/2.1

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

mj12bot/v1.4.8

Rule Path
Disallow /

facebookexternalhit/1.1

Rule Path
Disallow /

dalvik/2.1.0

Rule Path
Disallow /

feedburner/1.0

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

applebot/0.1

Rule Path
Disallow /

yandeximages/3.0

Rule Path
Disallow /

linkedinbot/1.0

Rule Path
Disallow /

semrushbot/7~bl

Rule Path
Disallow /

yandeximages/3.0

Rule Path
Disallow /

bingbot/2.0

Rule Path
Disallow /