Log In Sign Up

tops.lk
robots.txt

Robots Exclusion Standard data for tops.lk

Resource Scan

Scan Details

Site Domain tops.lk
Base Domain tops.lk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2025-12-29T03:51:50+00:00
Next Scan 2026-03-29T03:51:50+00:00

Last Successful Scan

Scanned2025-06-02T09:48:39+00:00
URL https://tops.lk/robots.txt
Redirect https://www.topssrilanka.com/robots.txt
Redirect Domain www.topssrilanka.com
Redirect Base topssrilanka.com
Domain IPs 104.21.49.251, 172.67.197.220, 2606:4700:3030::6815:31fb, 2606:4700:3034::ac43:c5dc
Redirect IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Response IP 104.21.96.1
Found Yes
Hash f40aee9a2264716f32b66e746b94df5d57aae7a5acd7cb1f54d587b786df855c
SimHash 2a5344c2cb13

Groups

mediapartners-google

Rule Path
Disallow

mozilla/3.0 (compatible;miner;mailto:miner@miner.com.br)

Rule Path
Disallow /

webferret

Rule Path
Disallow /

due to a deficiency in java it's not currently possible to set the user-agent.

Rule Path
Disallow /

no

Rule Path
Disallow /

arachnophilia

Rule Path
Disallow /

architextspider

Rule Path
Disallow /

aspider/0.09

Rule Path
Disallow /

auresys/1.0

Rule Path
Disallow /

backrub/*.*

Rule Path
Disallow /

big brother

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

bspider/1.0 libwww-perl/0.40

Rule Path
Disallow /

cactvs chemistry spider

Rule Path
Disallow /

digimarc cgireader/1.0

Rule Path
Disallow /

checkbot/x.xx lwp/5.x

Rule Path
Disallow /

cmc/0.01

Rule Path
Disallow /

combine/0.0

Rule Path
Disallow /

conceptbot/0.3

Rule Path
Disallow /

crescent internet toolpak http ole control v.1.0

Rule Path
Disallow /

root/0.1

Rule Path
Disallow /

cs-hkust-indexserver/1.0

Rule Path
Disallow /

cyberspyder/2.1

Rule Path
Disallow /

deweb/1.01

Rule Path
Disallow /

dragonbot/1.0 libwww/5.0

Rule Path
Disallow /

eit-link-verifier-robot/0.2

Rule Path
Disallow /

emacs-w3/v[0-9\.]+

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emc spider

Rule Path
Disallow /

explorersearch

Rule Path
Disallow /

explorer

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

felixide/1.0

Rule Path
Disallow /

hazel's ferret web hopper,

Rule Path
Disallow /

esirover v1.0

Rule Path
Disallow /

fido/0.9 harvest/1.4.pl2

Rule Path
Disallow /

hämähäkki/0.2

Rule Path
Disallow /

kit-fireball/2.0 libwww/5.0a

Rule Path
Disallow /

fish-search-robot

Rule Path
Disallow /

mozilla/2.0 (compatible fouineur v2.0; fouineur.9bit.qc.ca)

Rule Path
Disallow /

robot du crim 1.0a

Rule Path
Disallow /

freecrawl

Rule Path
Disallow /

funnelweb-1.0

Rule Path
Disallow /

gcreep/1.0

Rule Path
Disallow /

geturl.rexx v1.05

Rule Path
Disallow /

golem/1.1

Rule Path
Disallow /

gromit/1.0

Rule Path
Disallow /

gulliver/1.1

Rule Path
Disallow /

yes

Rule Path
Disallow /

aitcsrobot/1.1

Rule Path
Disallow /

*

Rule Path
Allow /
Disallow /admin.php
Disallow /admin/
Disallow /includes/
Disallow /blocks/
Disallow /modules/
Disallow /language/
Disallow /cgi-bin/
Disallow /shared/
Allow /shared/fonts/
Back to top

Other Records

Field Value
sitemap http://www.tops.lk/downloadsmap.xml
sitemap http://www.tops.lk/forumssitemap.xml
sitemap http://www.tops.lk/googlenewsmap.xml
sitemap http://www.tops.lk/googlesitemap.xml
sitemap http://www.tops.lk/googlespotnewsmap.xml
sitemap http://www.tops.lk/latestforumtopicsmap.xml
sitemap http://www.tops.lk/newsmap.xml
sitemap http://www.tops.lk/spotnewsmap.xml
sitemap http://www.tops.lk/spotubemap.xml
sitemap http://www.tops.lk/spotubevideomap.xml
sitemap http://www.tops.lk/tvrstreamfeed.xml
Back to top

Warnings

  • 4 invalid lines.
Back to top