Log In Sign Up

vans.com.sg
robots.txt

Robots Exclusion Standard data for vans.com.sg

Resource Scan

Scan Details

Site Domain vans.com.sg
Base Domain vans.com.sg
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-02T17:16:58+00:00
Next Scan 2024-12-01T17:16:58+00:00

Last Successful Scan

Scanned2023-10-16T10:51:00+00:00
URL https://www.vans.com.sg/robots.txt
Domain IPs 13.228.180.215, 54.254.80.156
Response IP 13.228.180.215
Found Yes
Hash 47e77dd0517d768ef75d4d03c216b570ac5d0a12ae81c22ceeb9f9a70a29b45f
SimHash 4cf0771890a7

Groups

*

Rule Path
Allow /$
Allow /*.html*
Disallow /*catalogsearch*/
Disallow /catalogsearch/
Disallow /*/onestepcheckout/*
Disallow /*/checkout/onepage/*
Disallow /*/customer/account/login*
Disallow /*/wishlist/index/add*
Disallow /*/enable-cookies*
Disallow /*/catalog/product_compare/add*
Disallow /*/reviews/index/write*
Disallow /*/facebook/customer_account/*
Disallow /*/sales/order/history
Disallow /*/customer/account
Disallow /*/wishlist
Disallow /*/customer/account/logout
Disallow /lib/
Disallow /*.php$
Disallow /pkginfo/
Disallow /report/
Disallow /var/
Disallow /catalog/
Disallow /customer/
Disallow /sendfriend/
Disallow /review/
Disallow /*SID%3D
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /*filtered*
Disallow /*ajax*
Disallow /lib/
Disallow /*.php$
Disallow /pkginfo/
Disallow /report/
Disallow /var/
Disallow /catalog/
Disallow /customer/
Disallow /sendfriend/
Disallow /review/
Disallow /*SID%3D
Disallow /*?
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /pub/
Disallow /tag/
Disallow /review/
Disallow /composer.json
Disallow /composer.lock
Disallow /CONTRIBUTING.md
Disallow /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow /COPYING.txt
Disallow /Gruntfile.js
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /nginx.conf.sample
Disallow /package.json
Disallow /php.ini.sample
Disallow /RELEASE_NOTES.txt
Disallow /*?*product_list_mode=
Disallow /*?*product_list_order=
Disallow /*?*product_list_limit=
Disallow /*?*product_list_dir=
Disallow /*.git
Disallow /*.CVS
Disallow /*.Zip$
Disallow /*.Svn$
Disallow /*.Idea$
Disallow /*.Sql$
Disallow /*.Tgz$

sistrix

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

jobs.de-robot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

seodiver

Rule Path
Disallow /

spbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

obot

Rule Path
Disallow /

fr-crawler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

cloudservermarketspider

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

lipperhey-kaus-australis

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

linkstats

Rule Path
Disallow /

jobboersebot

Rule Path
Disallow /

iccrawler

Rule Path
Disallow /

plista

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

lipperhey-kaus-australis

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

um-ic

Rule Path
Disallow /

mindupbot

Rule Path
Disallow /

sg-orbiter

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

kraken

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

openhosebot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Allow /

thumbsniper

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

implisensebot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

adscanner

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

python/3.5 aiohttp

Rule Path
Disallow /

toweya.com

Rule Path
Disallow /

netestate

Rule Path
Disallow /

bubing

Rule Path
Disallow /

linguee

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

indeedbot

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

gosign-security-crawler

Rule Path
Disallow /

siteliner

Rule Path
Disallow /

sabsimbot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

mb2345browser

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

zh-cn

Rule Path
Disallow /

micromessenger

Rule Path
Disallow /

zh_cn

Rule Path
Disallow /

kinza

Rule Path
Disallow /

datanyze

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

spaziodati

Rule Path
Disallow /

oppo\sa33

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

kinza

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

mb2345browser

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

*

Rule Path
Disallow /women/clothing/.html
Disallow /anaheim-factory-sid-dx.html
Disallow /women/featured/suede-zip.html
Disallow /anaheim-oldskool.html
Disallow /valentine.html
Disallow /men/shoes/sandals.html
Disallow /corecho-collection.html
Disallow /wheres-waldo.html
Disallow /retro-sport.html
Disallow /womens-diy.html
Disallow /sid.html
Disallow /boy-of-summer.html
Disallow /color-mix.html
Disallow /ultrarange-hi-mte.html
Disallow /diy-pack.html
Disallow /woven-platform.html
Disallow /anaheim-mixedprint.html
Disallow /men/featured/drill-chore-coat.html
Disallow /u-color.html
Disallow /libertyfabrics.html
Disallow /paisley-bandana.html
Disallow /tysonpeterson.html
Disallow /meadow-patchwork.html
Disallow /pigsuede.html
Disallow /andrew-reynolds.html
Disallow /anaheim-sk8-hi.html
Disallow /neighborhood.html
Disallow /women/featured/vans-crayola.html
Disallow /men/shoes/surf.html
Disallow /lizzie-armanto.html
Disallow /flour-shop.html
Disallow /women/accessories/sunglasses.html
Disallow /national-geographic-collection.html
Disallow /kyle-walker-pro-collection.html
Disallow /paisley.html
Disallow /skate-classics.html
Disallow /anaheim-factory-panda.html
Disallow /se-bikes.html
Disallow /more/more-products/surf.html
Disallow /c2h4.html
Disallow /quasi.html
Disallow /women/clothing/denim.html
Disallow /more/more-fun/girl-skate.html
Disallow /bmx-cult.html
Disallow /customculture.html
Disallow /patchwork-floral.html
Disallow /twisted.html
Disallow /all-style36.html
Disallow /liberaiders.html
Disallow /classic-sport-style-36.html
Disallow /kide-collection.html
Disallow /kazuki-kuraishi.html
Disallow /men/accessories/keychain.html
Disallow /chris-johanson.html
Disallow /vans-moma-collection.html
Disallow /tianran-collection.html
Disallow /anaheim-factory-leather-check.html
Disallow /all-platform.html

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 8
Back to top

Comments

  • www.robotstxt.org/
  • www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
  • Disable checkout & customer account
  • Disable Search pages
  • Disable common folders
  • Disable Tag & Review (Avoid duplicate content)
  • Common files
  • Disable sorting (Avoid duplicate content)
  • Disable version control folders and others
  • Disallow: Sistrix
  • Disallow: Sistrix
  • Disallow: Sistrix
  • Disallow: SEOkicks-Robot
  • Disallow: jobs.de-Robot
  • Backlink Analysis
  • Bot der Leipziger Unister Holding GmbH
  • http://www.opensiteexplorer.org/dotbot
  • http://www.searchmetrics.com
  • http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
  • http://www.domaintools.com/webmasters/surveybot.php
  • http://www.seodiver.com/bot
  • http://openlinkprofiler.org/bot
  • http://www.wotbox.com/bot/
  • http://www.meanpath.com/meanpathbot.html
  • http://www.backlinktest.com/crawler.html
  • http://www.brandwatch.com/magpie-crawler/
  • http://filterdb.iss.net/crawler/
  • http://webmeup-crawler.com
  • https://megaindex.com/crawler
  • http://www.cloudservermarket.com
  • http://www.trendiction.de/de/publisher/bot
  • http://www.exalead.com
  • http://www.career-x.de/bot.html
  • https://www.lipperhey.com/en/about/
  • https://www.lipperhey.com/en/about/
  • https://turnitin.com/robot/crawlerinfo.html
  • http://help.coccoc.com/
  • ubermetrics-technologies.com
  • datenbutler.de
  • http://searchgears.de/uber-uns/crawling-faq.html
  • http://commoncrawl.org/faq/
  • https://www.qwant.com/
  • http://linkfluence.net/
  • http://www.botje.com/plukkie.htm
  • https://www.safedns.com/searchbot
  • http://www.haosou.com/help/help_3_2.html
  • http://www.haosou.com/help/help_3_2.html
  • http://www.moz.com/dp/rogerbot
  • http://www.openhose.org/bot.html
  • http://www.screamingfrog.co.uk/seo-spider/
  • http://thumbsniper.com
  • http://www.radian6.com/crawler
  • http://cliqz.com/company/cliqzbot
  • https://www.aihitdata.com/about
  • http://www.trendiction.com/en/publisher/bot
  • http://seocompany.store
  • https://github.com/yasserg/crawler4j/
  • http://warebay.com/bot.html
  • http://www.website-datenbank.de/
  • http://law.di.unimi.it/BUbiNG.html
  • http://www.linguee.com/bot; bot@linguee.com
  • https://www.semrush.com/bot/
  • www.sentibot.eu
  • http://velen.io
  • https://moz.com/help/guides/moz-procedures/what-is-rogerbot
  • http://www.garlik.com
  • https://www.gosign.de/typo3-extension/typo3-sicherheitsmonitor/
  • http://www.siteliner.com/bot
  • https://sabsim.com
  • http://ltx71.com/
  • Chinese Bots
  • Slow down bots
Back to top

Warnings

  • 2 invalid lines.
Back to top