thestar.ke
robots.txt

Robots Exclusion Standard data for thestar.ke

Resource Scan

Scan Details

Site Domain thestar.ke
Base Domain thestar.ke
Scan Status Ok
Last Scan2024-05-07T16:50:46+00:00
Next Scan 2024-05-14T16:50:46+00:00

Last Scan

Scanned2024-05-07T16:50:46+00:00
URL https://thestar.ke/robots.txt
Redirect https://www.the-star.co.ke/robots.txt
Redirect Domain www.the-star.co.ke
Redirect Base the-star.co.ke
Domain IPs 2001:4860:4802:32::15, 2001:4860:4802:34::15, 2001:4860:4802:36::15, 2001:4860:4802:38::15, 216.239.32.21, 216.239.34.21, 216.239.36.21, 216.239.38.21
Redirect IPs 104.21.52.118, 172.67.198.128, 2606:4700:3035::6815:3476, 2606:4700:3036::ac43:c680
Response IP 104.21.52.118
Found Yes
Hash 3a2a6519dc0a34d9b590a67539d1fb65d8c0f8c21f766867b3a2ef2dbf5a2825
SimHash b69471cbcf41

Groups

*

Rule Path
Allow /classifieds/*.js
Allow /classifieds/*.css
Allow /classifieds/*.jpg
Allow /classifieds/*?browse=
Disallow /classifieds/*itemindex*
Disallow /classifieds/SaveAd/
Disallow /classifieds/confirm-search/
Disallow /classifieds/get-filter-results/
Disallow /classifieds/get-top-box-results/
Disallow /classifieds/login-saved-searches/
Disallow /classifieds/manage-saved-searches/
Disallow /classifieds/recover-password/
Disallow /classifieds/register-saved-searches/
Disallow /classifieds/SavedSearch/
Disallow /classifieds/ss-logout/
Disallow /classifieds/Subscriptions/
Disallow /classifieds/update-frequency/
Disallow /classifieds/update-password/
Disallow /classifieds/Account/
Disallow /classifieds/SingleAdPage/
Disallow /classifieds/SingleAdPage/Home/
Disallow /classifieds/SingleAdPage/Product/
Disallow /classifieds/SingleAdPage/Similar/
Disallow /classifieds/*ajax.svc*
Disallow /classifieds/*_nif
Disallow /classifieds/admanagement/*
Disallow /classifieds/Controls/*
Disallow /classifieds/controls/*
Disallow /classifieds/data/*
Disallow /classifieds/external-login/
Disallow /classifieds/ExternalApi/
Disallow /classifieds/get-results/*
Disallow /classifieds/JsonProvider/Components
Disallow /classifieds/mobile/search/
Disallow /classifieds/Report/
Disallow /classifieds/Rss/
Disallow /classifieds/set-admin-cookie/
Disallow /classifieds/SitemapHandler/
Disallow /classifieds/PartnerIntegration/
Disallow /classifieds/XmlProvider/
Disallow /classifieds/api/
Disallow /classifieds/*.aspx
Disallow /classifieds/Reviews/
Disallow /classifieds/serp/

ahrefsbot

Rule Path
Disallow /classifieds/

blexbot

Rule Path
Disallow /classifieds/

bloglines/3.1

Rule Path
Disallow /classifieds/

cityreview

Rule Path
Disallow /classifieds/

doc

Rule Path
Disallow /classifieds/

dotbot

Rule Path
Disallow /classifieds/

download ninja

Rule Path
Disallow /classifieds/

exabot

Rule Path
Disallow /classifieds/

fetch

Rule Path
Disallow /classifieds/

grapeshot

Rule Path
Disallow /classifieds/

grub-client

Rule Path
Disallow /classifieds/

httrack

Rule Path
Disallow /classifieds/

jyxobot/1

Rule Path
Disallow /classifieds/

k2spider

Rule Path
Disallow /classifieds/

larbin

Rule Path
Disallow /classifieds/

libwww

Rule Path
Disallow /classifieds/

linko

Rule Path
Disallow /classifieds/

microsoft.url.control

Rule Path
Disallow /classifieds/

msiecrawler

Rule Path
Disallow /classifieds/

npbot

Rule Path
Disallow /classifieds/

offline explorer

Rule Path
Disallow /classifieds/

petalbot

Rule Path
Disallow /classifieds/

proximic

Rule Path
Disallow /classifieds/

psbot

Rule Path
Disallow /classifieds/

semrushbot

Rule Path
Disallow /classifieds/

sitecheck.internetseer.com

Rule Path
Disallow /classifieds/

sitesnagger

Rule Path
Disallow /classifieds/

speedy

Rule Path
Disallow /classifieds/

teleport

Rule Path
Disallow /classifieds/

teleportpro

Rule Path
Disallow /classifieds/

ubicrawler

Rule Path
Disallow /classifieds/

webcopier

Rule Path
Disallow /classifieds/

webreaper

Rule Path
Disallow /classifieds/

webstripper

Rule Path
Disallow /classifieds/

webzip

Rule Path
Disallow /classifieds/

wget

Rule Path
Disallow /classifieds/

xenu

Rule Path
Disallow /classifieds/

zao

Rule Path
Disallow /classifieds/

zealbot

Rule Path
Disallow /classifieds/

zyborg

Rule Path
Disallow /classifieds/