Log In Sign Up

verkkokauppa.fi
robots.txt

Robots Exclusion Standard data for verkkokauppa.fi

Resource Scan

Scan Details

Site Domain verkkokauppa.fi
Base Domain verkkokauppa.fi
Scan Status Ok
Last Scan2024-06-04T12:31:42+00:00
Next Scan 2024-06-18T12:31:42+00:00

Last Scan

Scanned2024-06-04T12:31:42+00:00
URL https://verkkokauppa.fi/robots.txt
Redirect https://www.verkkokauppa.com/robots.txt
Redirect Domain www.verkkokauppa.com
Redirect Base verkkokauppa.com
Domain IPs 34.95.73.242
Redirect IPs 104.18.33.183, 172.64.154.73, 2606:4700:4400::6812:21b7, 2606:4700:4400::ac40:9a49
Response IP 104.18.33.183
Found Yes
Hash f48e7eec8618d6352abdcbe6fd836ed9a82a8aa7c6473667fcee7b3f571caa75
SimHash a218515beee4

Groups

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

fast

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

*

Rule Path
Disallow /fi/account/
Disallow /fi/orders/
Disallow /fi/tili/
Disallow /fi/checkout/
Disallow /fi/payment/
Disallow /fi/unsubscribe/
Disallow /ext/
Disallow /api/
Disallow /payment/
Disallow /pimg/
Disallow /resp-api/
Disallow /unsubscribe/
Disallow /fi/s$
Disallow /fi/s?

googlebot

Rule Path
Disallow /fi/product/*
Disallow /fi/reviews/*
Allow /fi/product/*/
Allow /fi/reviews/*/
Disallow /fi/outlet/yksittaiskappaleet/*
Disallow /fi/outlet/yksittaiskappaleet?
Back to top

Other Records

Field Value
sitemap https://www.verkkokauppa.com/gsitemaps1/sitemap.xml
Back to top

Comments

  • ####################
  • ##################
  • #################
  • ###############
  • #############
  • ###########
  • #########
  • ########
  • #####
  • ####
  • ##
  • VUODESTA 1992
  • Crawlers that are kind enough to obey, but which we'd rather not have
  • unless they're feeding search engines.
  • Some bots are known to be trouble, particularly those designed to copy
  • entire sites. Please obey robots.txt.
  • Misbehaving: requests much too fast:
  • Sorry, wget in its recursive mode is a frequent problem.
  • Please read the man page and use it properly; there is a
  • --wait option you can use to set the delay between hits,
  • for instance.
  • The 'grub' distributed client has been *very* poorly behaved.
  • Doesn't follow robots.txt anyway, but...
  • Hits many times per second, not acceptable
  • http://www.nameprotect.com/botinfo.html
  • A capture bot, downloads gazillions of pages with no public benefit
  • http://www.webreaper.net/
  • Block Superhaku of the old days
  • These are probably unnecessary now that we have proper canonicals, but will
  • do no harm (although the product page urls will change in the future)
  • Disallow pages with only the product number
  • Allow the ones with more than that
  • Disallow outlet product modal and outlet search result pages
  • Hint to robots where the sitemap is
Back to top