thestandard.com.hk
robots.txt

Robots Exclusion Standard data for thestandard.com.hk

Archived Snapshots

Resource Scan

Scan Details

Site Domain	thestandard.com.hk
Base Domain	thestandard.com.hk
Scan Status	Ok
Last Scan	2024-07-06T15:30:34+00:00
Next Scan	2024-07-13T15:30:34+00:00

Last Scan

Scanned	2024-07-06T15:30:34+00:00
URL	https://thestandard.com.hk/robots.txt
Redirect	https://www.thestandard.com.hk/robots.txt
Redirect Domain	www.thestandard.com.hk
Redirect Base	thestandard.com.hk
Domain IPs	104.22.78.203, 104.22.79.203, 172.67.43.132, 2606:4700:10::6816:4ecb, 2606:4700:10::6816:4fcb, 2606:4700:10::ac43:2b84
Redirect IPs	104.22.78.203, 104.22.79.203, 172.67.43.132, 2606:4700:10::6816:4ecb, 2606:4700:10::6816:4fcb, 2606:4700:10::ac43:2b84
Response IP	104.22.79.203
Found	Yes
Hash	57c746e67ca5a1a13a8518c5abca8f15f3fa426a8f409656ccbd5e391217951f
SimHash	4a1ed231e0a1

Groups

facebookexternalhit

Rule	Path
Allow	/

Rule

Path

Allow

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

facebookexternalhit/*

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

googlebot

Rule	Path
Allow	/

Rule

Path

Allow

slurp

Rule	Path
Allow	/

Rule

Path

Allow

msnbot

Rule	Path
Allow	/

Rule

Path

Allow

applebot

Rule	Path
Allow	/

Rule

Path

Allow

baiduspider

Rule	Path
Allow	/

Rule

Path

Allow

bingbot

Rule	Path
Allow	/

Rule

Path

Allow

facebot

Rule	Path
Allow	/

Rule

Path

Allow

twitterbot

Rule	Path
Allow	/

Rule

Path

Allow

blp_bbot

Rule	Path
Disallow	/

Rule

Path

Disallow

blp_bbot/0.1

Rule	Path
Disallow	/

Rule

Path

Disallow

blp_bbot/0.1

Rule	Path
Disallow	/

Rule

Path

Disallow

moodlebot

Rule	Path
Disallow	/

Rule

Path

Disallow

moodlebot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

eyemonit_bot_version_0.1_(http://www.eyemon.it/)

Rule	Path
Disallow	/

Rule

Path

Disallow

eyemonit_bot

Rule	Path
Disallow	/

Rule

Path

Disallow

fullstorybot/1.0
fullstorybot/1.0 (+https://www.fullstory.com)

Rule	Path
Disallow	/

Rule

Path

Disallow

gsa-crawler+(enterprise;+t3-hr8mk3s756etj;+google-support@extended-content.com)

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

moatbot

Rule	Path
Disallow	/

Rule

Path

Disallow

aaabot

Rule	Path
Disallow	/

Rule

Path

Disallow

anderspinkbot

Rule	Path
Disallow	/

Rule

Path

Disallow

seekport

Rule	Path
Disallow	/

Rule

Path

Disallow

voluumdsp-content-bot

Rule	Path
Disallow	/

Rule

Path

Disallow

special_archiver

Rule	Path
Disallow	/

Rule

Path

Disallow

megaindex.ru

Rule	Path
Disallow	/

Rule

Path

Disallow

Comments

User-agent: *
Disallow: /
BLP_bbot
BLP_bbot/0.1 bot
AhrefsBot
moatbot

thestandard.com.hkrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

facebookexternalhit

Other Records

facebookexternalhit/*

Other Records

googlebot

slurp

msnbot

applebot

baiduspider

bingbot

facebot

twitterbot

blp_bbot

blp_bbot/0.1

blp_bbot/0.1

moodlebot

moodlebot/1.0

eyemonit_bot_version_0.1_(http://www.eyemon.it/)

eyemonit_bot

fullstorybot/1.0fullstorybot/1.0 (+https://www.fullstory.com)

gsa-crawler+(enterprise;+t3-hr8mk3s756etj;+google-support@extended-content.com)

mj12bot

dotbot

ccbot

ahrefsbot

moatbot

aaabot

anderspinkbot

seekport

voluumdsp-content-bot

special_archiver

megaindex.ru

Comments

thestandard.com.hk
robots.txt

fullstorybot/1.0
fullstorybot/1.0 (+https://www.fullstory.com)