Log In Sign Up

hk.discount.wave-base.com
robots.txt

Robots Exclusion Standard data for hk.discount.wave-base.com

Resource Scan

Scan Details

Site Domain hk.discount.wave-base.com
Base Domain wave-base.com
Scan Status Ok
Last Scan2024-05-25T13:37:44+00:00
Next Scan 2024-06-24T13:37:44+00:00

Last Scan

Scanned2024-05-25T13:37:44+00:00
URL https://hk.discount.wave-base.com/robots.txt
Domain IPs 108.139.10.110, 108.139.10.121, 108.139.10.25, 108.139.10.27, 2600:9000:234b:1c00:d:47fe:7bc0:93a1, 2600:9000:234b:2e00:d:47fe:7bc0:93a1, 2600:9000:234b:3200:d:47fe:7bc0:93a1, 2600:9000:234b:3a00:d:47fe:7bc0:93a1, 2600:9000:234b:8e00:d:47fe:7bc0:93a1, 2600:9000:234b:ae00:d:47fe:7bc0:93a1, 2600:9000:234b:d200:d:47fe:7bc0:93a1, 2600:9000:234b:d400:d:47fe:7bc0:93a1
Response IP 108.157.60.21
Found Yes
Hash eb34f827e0b176418079c9e733137edf0872f8e1bbf8fffcfd0b06f994bda194
SimHash bb425543c960

Groups

*

Rule Path
Disallow /26297917/
Disallow /msg-pic/

barkrowler

Rule Path
Disallow /

babbar

Rule Path
Disallow /

muckrack

Rule Path
Disallow /

criteobot

Rule Path
Disallow /

censysinspect

Rule Path
Disallow /

cincraw

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

unknown

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

bytedance

Rule Path
Disallow /

spider-feedback@bytedance.com

Rule Path
Disallow /

bytedance.com

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

geedobot

Rule Path
Disallow /

netestate ne crawler (+http://www.website-datenbank.de/)

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

semrushbot-coub

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

new-sogou-spider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sogou mobile spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

sogou pic spider

Rule Path
Disallow /

sogou head spider

Rule Path
Disallow /

sogou orion spider

Rule Path
Disallow /

sogou-test-spider

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou pic agent

Rule Path
Disallow /

sogou spider2

Rule Path
Disallow /

sogou blog

Rule Path
Disallow /

sogou news spider

Rule Path
Disallow /

sogou orion spider

Rule Path
Disallow /

chinasospider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

proximic

Rule Path
Disallow /

mozilla/5.0 (compatible;picmole/1.0 +http://www.picmole.com)

Rule Path
Disallow /

lexxebot/1.0

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

mozilla/5.0 (compatible; spbot/2.0; http://www.seoprofiler.com/bot/ )

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

sosospider+(+http://help.soso.com/webspider.htm)

Rule Path
Disallow /

sitebot/0.1

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

crystalsemanticsbot

Rule Path
Disallow /

crystalsemanticsbot

Rule Path
Disallow /

netseer crawler

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

lexxebot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

discobot

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm

Product Comment
sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm 07)
Rule Path
Disallow /

sistrix

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

garlikcrawler/1.1 (http://garlik.com/, crawler@garlik.com)

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

nerdbynature.bot

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.0; windows nt; digext; dts agent

Rule Path
Disallow /

psbot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

addthis.com robot tech.support@clearspring.com

Rule Path
Disallow /

addthis.com

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

proximic

Rule Path
Disallow /

discoverybot

Rule Path
Disallow /

bl.uk_lddc_bot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

bender

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

yasni

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

exabot

Rule Path
Disallow /

pixray-seeker

Rule Path
Disallow /

linguee

Rule Path
Disallow /

integromedb

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

wesee:search

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

admantx

Rule Path
Disallow /

spbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

sogou develop spider

Rule Path
Disallow /

sogou head spider

Rule Path
Disallow /

sogou js robot

Rule Path
Disallow /

sogou orion spider

Rule Path
Disallow /

sogou pic agent

Rule Path
Disallow /

sogou pic spider

Rule Path
Disallow /

sogou push spider

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou-test-spider

Rule Path
Disallow /

powermapper

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

uptimerobot/2.0

Rule Path
Disallow /

ezooms robot

Rule Path
Disallow /

perl lwp

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

netestate ne crawler (+http://www.website-datenbank.de/)

Rule Path
Disallow /

wiseguys robot

Rule Path
Disallow /

turnitin robot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

turnitin bot

Rule Path
Disallow /

turnitinbot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)

Rule Path
Disallow /

turnitinbot/3.0

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

pimonster

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

eccp/1.0 (search@eniro.com)

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

gsa-crawler (enterprise; t4-knhh62cdkc2w3; gsa_manage@nikon-sys.co.jp)

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

*

Rule Path
Disallow /images/keep-shopping-and-enjoy-discounts.png
Back to top

Comments

  • Disallow: /doc/
  • Crawl-delay: 1
  • To block Coccocbot from crawling:
  • To block Dotbot from crawling:
  • To block Common Crawl Bot from crawling:
  • To block Common Crawl Bot from crawling:
  • To block GeedoBot from crawling:
  • Block netEstate NE Crawler (+http://www.website-datenbank.de/)
  • To block PetalBot from crawling:
  • To block AhrefsBot from crawling:
  • Blocked SemrushBot since it is creating lot of requests
  • To block SemrushBot from crawling your site for different SEO and technical issues:
  • To block SemrushBot from crawling your site for Backlink Audit tool:
  • To block SemrushBot from crawling your site for On Page SEO Checker tool and similar tools:
  • To block SemrushBot from checking URLs on your site for SWA tool:
  • To block SemrushBot from crawling your site for Content Analyzer and Post Tracking tools:
  • To block SplitSignalBot from crawling your site for SplitSignal tool:
  • To block SemrushBot-COUB from crawling your site for Content Outline Builder tool:
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Block Etaospider
  • Sogou Bot Blocking
  • Block MJ12bot as it is just noise
  • Block Ahrefs
  • Block Sogou
  • Block SEOkicks
  • Block BlexBot
  • Block SISTRIX
  • Block Uptime robot
  • Block Ezooms Robot
  • Block Perl LWP
  • Block BlexBot
  • Block netEstate NE Crawler (+http://www.website-datenbank.de/)
  • Block WiseGuys Robot
  • Block Turnitin Robot
  • Block Heritrix
  • Block pricepi
  • Block Searchmetrics Bot
  • Block Eniro
  • Block SoGou
  • Block Youdao
  • Block Nikon JP Crawler
  • Block MegaIndex.ru
  • User-agent: Baiduspider
  • Disallow: /
  • Block YandexBot
Back to top

Warnings

  • 4 invalid lines.
Back to top