thestar.com.my
robots.txt

Robots Exclusion Standard data for thestar.com.my

Resource Scan

Scan Details

Site Domain thestar.com.my
Base Domain thestar.com.my
Scan Status Ok
Last Scan2024-04-27T07:37:41+00:00
Next Scan 2024-05-04T07:37:41+00:00

Last Scan

Scanned2024-04-27T07:37:41+00:00
URL https://thestar.com.my/robots.txt
Redirect https://www.thestar.com.my/robots.txt
Redirect Domain www.thestar.com.my
Redirect Base thestar.com.my
Domain IPs 18.139.211.27
Redirect IPs 18.155.68.100, 18.155.68.109, 18.155.68.16, 18.155.68.95
Response IP 18.155.68.109
Found Yes
Hash 6607ef9e9551ddfe1781d2646bfd86c524cb909fd63869dbda9b4a3477b02120
SimHash 6d847224c513

Groups

*

Rule Path
Disallow /bin/
Disallow /temp/
Disallow /themes/

ahrefsbot/6.1

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

cutbot;+1.5

Rule Path
Disallow /

cutbot/1.5

Rule Path
Disallow /

cutbot/+1.5

Rule Path
Disallow /

slurp

Rule Path
Disallow /

yahoo! slurp

Rule Path
Disallow /

toutiaospider

Rule Path
Disallow /

toutiaospider/1.0

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

baiduspider-render

Rule Path
Disallow /

baiduspider-render/2.0

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot/3~bl

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yandeximages/3.0

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

paperlibot/2.1

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou web spider/4.0

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

cliqzbot/2.0

Rule Path
Disallow /

voluumdsp-content-bot/2.0

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

ahrefsbot/5.2

Rule Path
Disallow /

ahrefsbot/6.1

Rule Path
Disallow /

freshbot/4.0

Rule Path
Disallow /