vans.com.hk
robots.txt

Robots Exclusion Standard data for vans.com.hk

Resource Scan

Scan Details

Site Domain vans.com.hk
Base Domain vans.com.hk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-05T17:09:11+00:00
Next Scan 2024-12-04T17:09:11+00:00

Last Successful Scan

Scanned2023-10-19T06:47:09+00:00
URL https://vans.com.hk/robots.txt
Domain IPs 13.215.16.144, 54.169.76.211
Response IP 13.215.16.144
Found Yes
Hash 47e77dd0517d768ef75d4d03c216b570ac5d0a12ae81c22ceeb9f9a70a29b45f
SimHash 4cf0771890a7

Groups

*

Rule Path
Allow /$
Allow /*.html*
Disallow /*catalogsearch*/
Disallow /catalogsearch/
Disallow /*/onestepcheckout/*
Disallow /*/checkout/onepage/*
Disallow /*/customer/account/login*
Disallow /*/wishlist/index/add*
Disallow /*/enable-cookies*
Disallow /*/catalog/product_compare/add*
Disallow /*/reviews/index/write*
Disallow /*/facebook/customer_account/*
Disallow /*/sales/order/history
Disallow /*/customer/account
Disallow /*/wishlist
Disallow /*/customer/account/logout
Disallow /lib/
Disallow /*.php$
Disallow /pkginfo/
Disallow /report/
Disallow /var/
Disallow /catalog/
Disallow /customer/
Disallow /sendfriend/
Disallow /review/
Disallow /*SID%3D
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /*filtered*
Disallow /*ajax*
Disallow /lib/
Disallow /*.php$
Disallow /pkginfo/
Disallow /report/
Disallow /var/
Disallow /catalog/
Disallow /customer/
Disallow /sendfriend/
Disallow /review/
Disallow /*SID%3D
Disallow /*?
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /pub/
Disallow /tag/
Disallow /review/
Disallow /composer.json
Disallow /composer.lock
Disallow /CONTRIBUTING.md
Disallow /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow /COPYING.txt
Disallow /Gruntfile.js
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /nginx.conf.sample
Disallow /package.json
Disallow /php.ini.sample
Disallow /RELEASE_NOTES.txt
Disallow /*?*product_list_mode=
Disallow /*?*product_list_order=
Disallow /*?*product_list_limit=
Disallow /*?*product_list_dir=
Disallow /*.git
Disallow /*.CVS
Disallow /*.Zip$
Disallow /*.Svn$
Disallow /*.Idea$
Disallow /*.Sql$
Disallow /*.Tgz$

sistrix

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

jobs.de-robot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

seodiver

Rule Path
Disallow /

spbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

obot

Rule Path
Disallow /

fr-crawler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

cloudservermarketspider

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

lipperhey-kaus-australis

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

linkstats

Rule Path
Disallow /

jobboersebot

Rule Path
Disallow /

iccrawler

Rule Path
Disallow /

plista

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

lipperhey-kaus-australis

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

um-ic

Rule Path
Disallow /

mindupbot

Rule Path
Disallow /

sg-orbiter

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

kraken

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

openhosebot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Allow /

thumbsniper

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

implisensebot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

adscanner

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

python/3.5 aiohttp

Rule Path
Disallow /

toweya.com

Rule Path
Disallow /

netestate

Rule Path
Disallow /

bubing

Rule Path
Disallow /

linguee

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

indeedbot

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

gosign-security-crawler

Rule Path
Disallow /

siteliner

Rule Path
Disallow /

sabsimbot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

mb2345browser

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

zh-cn

Rule Path
Disallow /

micromessenger

Rule Path
Disallow /

zh_cn

Rule Path
Disallow /

kinza

Rule Path
Disallow /

datanyze

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

spaziodati

Rule Path
Disallow /

oppo\sa33

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

kinza

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

mb2345browser

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

*

Rule Path
Disallow /women/clothing/.html
Disallow /anaheim-factory-sid-dx.html
Disallow /women/featured/suede-zip.html
Disallow /anaheim-oldskool.html
Disallow /valentine.html
Disallow /men/shoes/sandals.html
Disallow /corecho-collection.html
Disallow /wheres-waldo.html
Disallow /retro-sport.html
Disallow /womens-diy.html
Disallow /sid.html
Disallow /boy-of-summer.html
Disallow /color-mix.html
Disallow /ultrarange-hi-mte.html
Disallow /diy-pack.html
Disallow /woven-platform.html
Disallow /anaheim-mixedprint.html
Disallow /men/featured/drill-chore-coat.html
Disallow /u-color.html
Disallow /libertyfabrics.html
Disallow /paisley-bandana.html
Disallow /tysonpeterson.html
Disallow /meadow-patchwork.html
Disallow /pigsuede.html
Disallow /andrew-reynolds.html
Disallow /anaheim-sk8-hi.html
Disallow /neighborhood.html
Disallow /women/featured/vans-crayola.html
Disallow /men/shoes/surf.html
Disallow /lizzie-armanto.html
Disallow /flour-shop.html
Disallow /women/accessories/sunglasses.html
Disallow /national-geographic-collection.html
Disallow /kyle-walker-pro-collection.html
Disallow /paisley.html
Disallow /skate-classics.html
Disallow /anaheim-factory-panda.html
Disallow /se-bikes.html
Disallow /more/more-products/surf.html
Disallow /c2h4.html
Disallow /quasi.html
Disallow /women/clothing/denim.html
Disallow /more/more-fun/girl-skate.html
Disallow /bmx-cult.html
Disallow /customculture.html
Disallow /patchwork-floral.html
Disallow /twisted.html
Disallow /all-style36.html
Disallow /liberaiders.html
Disallow /classic-sport-style-36.html
Disallow /kide-collection.html
Disallow /kazuki-kuraishi.html
Disallow /men/accessories/keychain.html
Disallow /chris-johanson.html
Disallow /vans-moma-collection.html
Disallow /tianran-collection.html
Disallow /anaheim-factory-leather-check.html
Disallow /all-platform.html

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 8

Comments

  • www.robotstxt.org/
  • www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
  • Disable checkout & customer account
  • Disable Search pages
  • Disable common folders
  • Disable Tag & Review (Avoid duplicate content)
  • Common files
  • Disable sorting (Avoid duplicate content)
  • Disable version control folders and others
  • Disallow: Sistrix
  • Disallow: Sistrix
  • Disallow: Sistrix
  • Disallow: SEOkicks-Robot
  • Disallow: jobs.de-Robot
  • Backlink Analysis
  • Bot der Leipziger Unister Holding GmbH
  • http://www.opensiteexplorer.org/dotbot
  • http://www.searchmetrics.com
  • http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
  • http://www.domaintools.com/webmasters/surveybot.php
  • http://www.seodiver.com/bot
  • http://openlinkprofiler.org/bot
  • http://www.wotbox.com/bot/
  • http://www.meanpath.com/meanpathbot.html
  • http://www.backlinktest.com/crawler.html
  • http://www.brandwatch.com/magpie-crawler/
  • http://filterdb.iss.net/crawler/
  • http://webmeup-crawler.com
  • https://megaindex.com/crawler
  • http://www.cloudservermarket.com
  • http://www.trendiction.de/de/publisher/bot
  • http://www.exalead.com
  • http://www.career-x.de/bot.html
  • https://www.lipperhey.com/en/about/
  • https://www.lipperhey.com/en/about/
  • https://turnitin.com/robot/crawlerinfo.html
  • http://help.coccoc.com/
  • ubermetrics-technologies.com
  • datenbutler.de
  • http://searchgears.de/uber-uns/crawling-faq.html
  • http://commoncrawl.org/faq/
  • https://www.qwant.com/
  • http://linkfluence.net/
  • http://www.botje.com/plukkie.htm
  • https://www.safedns.com/searchbot
  • http://www.haosou.com/help/help_3_2.html
  • http://www.haosou.com/help/help_3_2.html
  • http://www.moz.com/dp/rogerbot
  • http://www.openhose.org/bot.html
  • http://www.screamingfrog.co.uk/seo-spider/
  • http://thumbsniper.com
  • http://www.radian6.com/crawler
  • http://cliqz.com/company/cliqzbot
  • https://www.aihitdata.com/about
  • http://www.trendiction.com/en/publisher/bot
  • http://seocompany.store
  • https://github.com/yasserg/crawler4j/
  • http://warebay.com/bot.html
  • http://www.website-datenbank.de/
  • http://law.di.unimi.it/BUbiNG.html
  • http://www.linguee.com/bot; bot@linguee.com
  • https://www.semrush.com/bot/
  • www.sentibot.eu
  • http://velen.io
  • https://moz.com/help/guides/moz-procedures/what-is-rogerbot
  • http://www.garlik.com
  • https://www.gosign.de/typo3-extension/typo3-sicherheitsmonitor/
  • http://www.siteliner.com/bot
  • https://sabsim.com
  • http://ltx71.com/
  • Chinese Bots
  • Slow down bots

Warnings

  • 2 invalid lines.