Log In Sign Up

ja.wikihow.com
robots.txt

Robots Exclusion Standard data for ja.wikihow.com

Resource Scan

Scan Details

Site Domain ja.wikihow.com
Base Domain wikihow.com
Scan Status Ok
Last Scan2024-04-25T14:24:44+00:00
Next Scan 2024-05-25T14:24:44+00:00

Last Scan

Scanned2024-04-25T14:24:44+00:00
URL https://ja.wikihow.com/robots.txt
Redirect https://www.wikihow.jp/robots.txt
Redirect Domain www.wikihow.jp
Redirect Base wikihow.jp
Domain IPs 151.101.1.91, 151.101.129.91, 151.101.193.91, 151.101.65.91
Redirect IPs 151.101.1.91, 151.101.129.91, 151.101.193.91, 151.101.65.91
Response IP 199.232.45.91
Found Yes
Hash eaf55b3d3a780395c143a5ff3a5e1b1b5631711d787310b642811fb65f882041
SimHash 74525115c4f5

Groups

anthropic-ai

Rule Path
Disallow /

archive.org

Rule Path
Disallow /api.php
Disallow /index.php
Disallow /Special%3A

ccbot

Rule Path
Disallow /

doc

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

fetch

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

hmse_robot

Rule Path
Disallow /

httrack

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

linko

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

npbot

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wget

Rule Path
Disallow /

xenu

Rule Path
Disallow /

zao

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

adsbot-google

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

googlebot

Rule Path
Allow /

*

Rule Path
Allow /Special%3ABlock
Allow /Special%3ABlockList
Allow /Special%3ACategoryListing
Allow /Special%3ACharity
Allow /Special%3AEmailUser
Allow /Special%3ALSearch
Allow /Special%3ANewPages
Allow /Special%3A%E6%9C%80%E6%96%B0%E9%A1%B5%E9%9D%A2
Allow /Special%3AQABox
Allow /Special%3ASearchAd
Allow /Special%3ASitemap
Allow /Special%3AThankAuthors
Allow /Special%3AUserLogin
Allow /%D8%AE%D8%A7%D8%B5%3ABlock
Allow /%D8%AE%D8%A7%D8%B5%3ABlockList
Allow /%D8%AE%D8%A7%D8%B5%3ACategoryListing
Allow /%D8%AE%D8%A7%D8%B5%3ACharity
Allow /%D8%AE%D8%A7%D8%B5%3AEmailUser
Allow /%D8%AE%D8%A7%D8%B5%3ALSearch
Allow /%D8%AE%D8%A7%D8%B5%3ANewPages
Allow /%D8%AE%D8%A7%D8%B5%3A%D8%B5%D9%81%D8%AD%D8%A7%D8%AA_%D8%AC%D8%AF%D9%8A%D8%AF%D8%A9
Allow /%D8%AE%D8%A7%D8%B5%3AQABox
Allow /%D8%AE%D8%A7%D8%B5%3ASearchAd
Allow /%D8%AE%D8%A7%D8%B5%3ASitemap
Allow /%D8%AE%D8%A7%D8%B5%3AThankAuthors
Allow /%D8%AE%D8%A7%D8%B5%3AUserLogin
Allow /Speci%C3%A1ln%C3%AD%3ABlock
Allow /Speci%C3%A1ln%C3%AD%3ABlockList
Allow /Speci%C3%A1ln%C3%AD%3ACategoryListing
Allow /Speci%C3%A1ln%C3%AD%3ACharity
Allow /Speci%C3%A1ln%C3%AD%3AEmailUser
Allow /Speci%C3%A1ln%C3%AD%3ALSearch
Allow /Speci%C3%A1ln%C3%AD%3ANewPages
Allow /Speci%C3%A1ln%C3%AD%3ANov%C3%A9_str%C3%A1nky
Allow /Speci%C3%A1ln%C3%AD%3AQABox
Allow /Speci%C3%A1ln%C3%AD%3ASearchAd
Allow /Speci%C3%A1ln%C3%AD%3ASitemap
Allow /Speci%C3%A1ln%C3%AD%3AThankAuthors
Allow /Speci%C3%A1ln%C3%AD%3AUserLogin
Allow /Spezial%3ABlock
Allow /Spezial%3ABlockList
Allow /Spezial%3ACategoryListing
Allow /Spezial%3ACharity
Allow /Spezial%3AEmailUser
Allow /Spezial%3ALSearch
Allow /Spezial%3ANewPages
Allow /Spezial%3ANeue_Seiten
Allow /Spezial%3AQABox
Allow /Spezial%3ASearchAd
Allow /Spezial%3ASitemap
Allow /Spezial%3AThankAuthors
Allow /Spezial%3AUserLogin
Allow /Especial%3ABlock
Allow /Especial%3ABlockList
Allow /Especial%3ACategoryListing
Allow /Especial%3ACharity
Allow /Especial%3AEmailUser
Allow /Especial%3ALSearch
Allow /Especial%3ANewPages
Allow /Especial%3AP%C3%A1ginasNuevas
Allow /Especial%3AP%C3%A1ginas_novas
Allow /Especial%3AQABox
Allow /Especial%3ASearchAd
Allow /Especial%3ASitemap
Allow /Especial%3AThankAuthors
Allow /Especial%3AUserLogin
Allow /Sp%C3%A9cial%3ABlock
Allow /Sp%C3%A9cial%3ABlockList
Allow /Sp%C3%A9cial%3ACategoryListing
Allow /Sp%C3%A9cial%3ACharity
Allow /Sp%C3%A9cial%3AEmailUser
Allow /Sp%C3%A9cial%3ALSearch
Allow /Sp%C3%A9cial%3ANewPages
Allow /Sp%C3%A9cial%3ANouvelles_pages
Allow /Sp%C3%A9cial%3AQABox
Allow /Sp%C3%A9cial%3ASearchAd
Allow /Sp%C3%A9cial%3ASitemap
Allow /Sp%C3%A9cial%3AThankAuthors
Allow /Sp%C3%A9cial%3AUserLogin
Allow /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3ABlock
Allow /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3ABlockList
Allow /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3ACategoryListing
Allow /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3ACharity
Allow /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3AEmailUser
Allow /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3ALSearch
Allow /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3ANewPages
Allow /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3A%E0%A4%A8%E0%A4%8F_%E0%A4%AA%E0%A5%83%E0%A4%B7%E0%A5%8D%E0%A4%A0
Allow /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3AQABox
Allow /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3ASearchAd
Allow /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3ASitemap
Allow /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3AThankAuthors
Allow /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3AUserLogin
Allow /Istimewa%3ABlock
Allow /Istimewa%3ABlockList
Allow /Istimewa%3ACategoryListing
Allow /Istimewa%3ACharity
Allow /Istimewa%3AEmailUser
Allow /Istimewa%3ALSearch
Allow /Istimewa%3ANewPages
Allow /Istimewa%3AHalaman_baru
Allow /Istimewa%3AQABox
Allow /Istimewa%3ASearchAd
Allow /Istimewa%3ASitemap
Allow /Istimewa%3AThankAuthors
Allow /Istimewa%3AUserLogin
Allow /Speciale%3ABlock
Allow /Speciale%3ABlockList
Allow /Speciale%3ACategoryListing
Allow /Speciale%3ACharity
Allow /Speciale%3AEmailUser
Allow /Speciale%3ALSearch
Allow /Speciale%3ANewPages
Allow /Speciale%3APaginePi%C3%B9Recenti
Allow /Speciale%3AQABox
Allow /Speciale%3ASearchAd
Allow /Speciale%3ASitemap
Allow /Speciale%3AThankAuthors
Allow /Speciale%3AUserLogin
Allow /%E7%89%B9%E5%88%A5%3ABlock
Allow /%E7%89%B9%E5%88%A5%3ABlockList
Allow /%E7%89%B9%E5%88%A5%3ACategoryListing
Allow /%E7%89%B9%E5%88%A5%3ACharity
Allow /%E7%89%B9%E5%88%A5%3AEmailUser
Allow /%E7%89%B9%E5%88%A5%3ALSearch
Allow /%E7%89%B9%E5%88%A5%3ANewPages
Allow /%E7%89%B9%E5%88%A5%3A%E6%96%B0%E3%81%97%E3%81%84%E3%83%9A%E3%83%BC%E3%82%B8
Allow /%E7%89%B9%E5%88%A5%3AQABox
Allow /%E7%89%B9%E5%88%A5%3ASearchAd
Allow /%E7%89%B9%E5%88%A5%3ASitemap
Allow /%E7%89%B9%E5%88%A5%3AThankAuthors
Allow /%E7%89%B9%E5%88%A5%3AUserLogin
Allow /%ED%8A%B9%EC%88%98%3ABlock
Allow /%ED%8A%B9%EC%88%98%3ABlockList
Allow /%ED%8A%B9%EC%88%98%3ACategoryListing
Allow /%ED%8A%B9%EC%88%98%3ACharity
Allow /%ED%8A%B9%EC%88%98%3AEmailUser
Allow /%ED%8A%B9%EC%88%98%3ALSearch
Allow /%ED%8A%B9%EC%88%98%3ANewPages
Allow /%ED%8A%B9%EC%88%98%3A%EC%83%88%EB%AC%B8%EC%84%9C
Allow /%ED%8A%B9%EC%88%98%3AQABox
Allow /%ED%8A%B9%EC%88%98%3ASearchAd
Allow /%ED%8A%B9%EC%88%98%3ASitemap
Allow /%ED%8A%B9%EC%88%98%3AThankAuthors
Allow /%ED%8A%B9%EC%88%98%3AUserLogin
Allow /Speciaal%3ABlock
Allow /Speciaal%3ABlockList
Allow /Speciaal%3ACategoryListing
Allow /Speciaal%3ACharity
Allow /Speciaal%3AEmailUser
Allow /Speciaal%3ALSearch
Allow /Speciaal%3ANewPages
Allow /Speciaal%3ANieuwePaginas
Allow /Speciaal%3AQABox
Allow /Speciaal%3ASearchAd
Allow /Speciaal%3ASitemap
Allow /Speciaal%3AThankAuthors
Allow /Speciaal%3AUserLogin
Allow /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3ABlock
Allow /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3ABlockList
Allow /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3ACategoryListing
Allow /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3ACharity
Allow /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3AEmailUser
Allow /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3ALSearch
Allow /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3ANewPages
Allow /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3A%D0%9D%D0%BE%D0%B2%D1%8B%D0%B5_%D1%81%D1%82%D1%80%D0%B0%D0%BD%D0%B8%D1%86%D1%8B
Allow /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3AQABox
Allow /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3ASearchAd
Allow /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3ASitemap
Allow /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3AThankAuthors
Allow /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3AUserLogin
Allow /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9%3ABlock
Allow /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9%3ABlockList
Allow /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9%3ACategoryListing
Allow /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9%3ACharity
Allow /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9%3AEmailUser
Allow /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9%3ALSearch
Allow /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9%3ANewPages
Allow /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9%3A%E0%B8%AB%E0%B8%99%E0%B9%89%E0%B8%B2%E0%B9%83%E0%B8%AB%E0%B8%A1%E0%B9%88
Allow /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9%3AQABox
Allow /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9%3ASearchAd
Allow /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9%3ASitemap
Allow /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9%3AThankAuthors
Allow /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9%3AUserLogin
Allow /%C3%96zel%3AT%C3%BCmSayfalar
Allow /%C3%96zel%3ABlock
Allow /%C3%96zel%3AEngelle
Allow /%C3%96zel%3ABlockList
Allow /%C3%96zel%3AEngelListesi
Allow /%C3%96zel%3ACategoryListing
Allow /%C3%96zel%3ACharity
Allow /%C3%96zel%3AEmailUser
Allow /%C3%96zel%3AE-postaG%C3%B6nder
Allow /%C3%96zel%3ALSearch
Allow /%C3%96zel%3ANewPages
Allow /%C3%96zel%3AYeniSayfalar
Allow /%C3%96zel%3AQABox
Allow /%C3%96zel%3ASearchAd
Allow /%C3%96zel%3ASitemap
Allow /%C3%96zel%3AThankAuthors
Allow /%C3%96zel%3AUserLogin
Allow /%C3%96zel%3AKullan%C4%B1c%C4%B1OturumuA%C3%A7ma
Allow /%C4%90%E1%BA%B7c_bi%E1%BB%87t%3ABlock
Allow /%C4%90%E1%BA%B7c_bi%E1%BB%87t%3ABlockList
Allow /%C4%90%E1%BA%B7c_bi%E1%BB%87t%3ACategoryListing
Allow /%C4%90%E1%BA%B7c_bi%E1%BB%87t%3ACharity
Allow /%C4%90%E1%BA%B7c_bi%E1%BB%87t%3AEmailUser
Allow /%C4%90%E1%BA%B7c_bi%E1%BB%87t%3ALSearch
Allow /%C4%90%E1%BA%B7c_bi%E1%BB%87t%3ANewPages
Allow /%C4%90%E1%BA%B7c_bi%E1%BB%87t%3ATrang_m%E1%BB%9Bi
Allow /%C4%90%E1%BA%B7c_bi%E1%BB%87t%3AQABox
Allow /%C4%90%E1%BA%B7c_bi%E1%BB%87t%3ASearchAd
Allow /%C4%90%E1%BA%B7c_bi%E1%BB%87t%3ASitemap
Allow /%C4%90%E1%BA%B7c_bi%E1%BB%87t%3AThankAuthors
Allow /%C4%90%E1%BA%B7c_bi%E1%BB%87t%3AUserLogin
Allow /%D9%88%DB%8C%DA%98%D9%87%3ABlock
Allow /%D9%88%DB%8C%DA%98%D9%87%3A%D8%A8%D8%B3%D8%AA%D9%86_%D9%86%D8%B4%D8%A7%D9%86%DB%8C_%D8%A2%DB%8C%E2%80%8C%D9%BE%DB%8C
Allow /%D9%88%DB%8C%DA%98%D9%87%3ABlockList
Allow /%D9%88%DB%8C%DA%98%D9%87%3A%D9%81%D9%87%D8%B1%D8%B3%D8%AA_%D8%A8%D8%B3%D8%AA%D9%87%E2%80%8C%D8%B4%D8%AF%D9%87%E2%80%8C%D9%87%D8%A7
Allow /%D9%88%DB%8C%DA%98%D9%87%3ACategoryListing
Allow /%D9%88%DB%8C%DA%98%D9%87%3ACharity
Allow /%D9%88%DB%8C%DA%98%D9%87%3AEmailUser
Allow /%D9%88%DB%8C%DA%98%D9%87%3A%D8%A7%DB%8C%D9%85%DB%8C%D9%84_%D8%A8%D9%87_%DA%A9%D8%A7%D8%B1%D8%A8%D8%B1
Allow /%D9%88%DB%8C%DA%98%D9%87%3ALSearch
Allow /%D9%88%DB%8C%DA%98%D9%87%3ANewPages
Allow /%D9%88%DB%8C%DA%98%D9%87%3A%D8%B5%D9%81%D8%AD%D9%87%E2%80%8C%D9%87%D8%A7%DB%8C_%D8%AA%D8%A7%D8%B2%D9%87
Allow /%D9%88%DB%8C%DA%98%D9%87%3AQABox
Allow /%D9%88%DB%8C%DA%98%D9%87%3ASearchAd
Allow /%D9%88%DB%8C%DA%98%D9%87%3ASitemap
Allow /%D9%88%DB%8C%DA%98%D9%87%3AThankAuthors
Allow /%D9%88%DB%8C%DA%98%D9%87%3AUserLogin
Allow /%D9%88%DB%8C%DA%98%D9%87%3A%D9%88%D8%B1%D9%88%D8%AF_%D8%A8%D9%87_%D8%B3%D8%A7%D9%85%D8%A7%D9%86%D9%87
Allow /index.php?*action=credits
Allow /index.php?*MathShowImage
Allow /index.php?*printable
Allow /index.php?title=*variant=
Disallow /index.php
Disallow /*feed%3Drss
Disallow /*action%3Ddelete
Disallow /*action%3Dhistory
Disallow /Special%3A
Disallow /%D8%AE%D8%A7%D8%B5%3A
Disallow /Speci%C3%A1ln%C3%AD%3A
Disallow /Spezial%3A
Disallow /Especial%3A
Disallow /Sp%C3%A9cial%3A
Disallow /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3A
Disallow /Istimewa%3A
Disallow /Speciale%3A
Disallow /%E7%89%B9%E5%88%A5%3A
Disallow /%ED%8A%B9%EC%88%98%3A
Disallow /Speciaal%3A
Disallow /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3A
Disallow /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9%3A
Disallow /%C3%96zel%3A
Disallow /%C4%90%E1%BA%B7c_bi%E1%BB%87t%3A
Disallow /%D9%88%DB%8C%DA%98%D9%87%3A
Disallow /Image%3A
Disallow /Imagen%3A
Disallow /Immagine%3A
Disallow /Bild%3A
Disallow /%E0%A4%9A%E0%A4%BF%E0%A4%A4%E0%A5%8D%E0%A4%B0%3A
Disallow /Afbeelding%3A
Disallow /Imagem%3A
Disallow /%D0%A4%D0%B0%D0%B9%D0%BB%3A
Disallow /File%3A
Disallow /Soubor%3A
Disallow /Berkas%3A
Disallow /Dosya%3A
Disallow /%E3%83%95%E3%82%A1%E3%82%A4%E3%83%AB%3A
Disallow /%D9%BE%D8%B1%D9%88%D9%86%D8%AF%D9%87%3A
Disallow /*platform%3D
Back to top

Comments

  • robots.txt for non-en lang domains
  • based on wikipedia.org's robots.txt
  • Note: contains /Special page prefixes for:
  • en, zh: /Special
  • ar: /%D8%AE%D8%A7%D8%B5
  • cs: /Speci%C3%A1ln%C3%AD
  • de: /Spezial
  • es, pt: /Especial
  • fa: /%D9%88%DB%8C%DA%98%D9%87
  • fr: /Sp%C3%A9cial
  • hi: /%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7
  • id: /Istimewa
  • it: /Speciale
  • ja: /%E7%89%B9%E5%88%A5
  • ko: /%ED%8A%B9%EC%88%98
  • nl: /Speciaal
  • ru: /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F
  • th: /%E0%B8%9E%E0%B8%B4%E0%B9%80%E0%B8%A8%E0%B8%A9
  • tr: /%C3%96zel
  • vi: /%C4%90%E1%BA%B7c_bi%E1%BB%87t
  • Our general guideline is not to crawl more than 1 page every 3 seconds.
  • If your bot supports such a thing using the 'Crawl-delay' or another
  • instruction, please let us know. We can add it to our robots.txt.
  • Friendly, low-speed bots are welcome viewing article pages, but not
  • dynamically-generated pages please. Article pages contain our site's
  • real content.
  • Doesn't follow robots.txt anyway, but...
  • Requests many pages per second
  • http://www.nameprotect.com/botinfo.html
  • Some bots are known to be trouble, particularly those designed to copy
  • entire sites. Please obey robots.txt.
  • A capture bot, downloads gazillions of pages with no public benefit
  • http://www.webreaper.net/
  • wget in recursive mode uses too many resources for us.
  • Please read the man page and use it properly; there is a
  • --wait option you can use to set the delay between hits,
  • for instance. Please wait 3 seconds between each request.
Back to top