Log In Sign Up

colnect.com
robots.txt

Robots Exclusion Standard data for colnect.com

Resource Scan

Scan Details

Site Domain colnect.com
Base Domain colnect.com
Scan Status Ok
Last Scan2024-05-23T02:34:46+00:00
Next Scan 2024-05-30T02:34:46+00:00

Last Scan

Scanned2024-05-23T02:34:46+00:00
URL https://colnect.com/robots.txt
Domain IPs 162.55.91.207
Response IP 162.55.91.207
Found Yes
Hash 86db1f56b95027f61036fe4ce3c415df4f7d158d6f929281af42d17f10026cde
SimHash 5011451865a2

Groups

facebookexternalhit

Rule Path
Disallow /

seznambot

Rule Path
Disallow *..
Disallow /teletalk/
Disallow /downloads/
Disallow /forums/download/
Disallow /main/set_language/
Disallow /gu
Disallow /ht
Disallow /kk
Disallow /ml
Disallow /mk
Disallow /pa/
Disallow /sw
Disallow /ta
Disallow /te
Allow /images
Disallow /*/self$
Disallow /*/self/
Disallow /*/new/
Disallow /*/online/
Disallow /*/edit/
Disallow /*/login$
Disallow /*/account$
Disallow /*/account/
Disallow /*/collectors/rate/
Disallow /*/collectors/log/
Disallow /*/collectors/list/friends/
Disallow /*/collectors/list/marked_as_friend/
Disallow /*/collectors/list/watchlist/
Disallow /*/collectors/list/best_matches/
Disallow /*/collectors/log_global/
Disallow /*/collectors/mark/
Disallow /*/collectors/my_profile
Disallow /*/collectors/edit_log
Disallow /*/collectors/translation_log
Disallow /*/directory/
Disallow /*/item/view_collectors/
Disallow /*/main/
Disallow /*/collectors/*/active/
Disallow /*/cart/
Disallow /*/seller/
Disallow /*/transaction/
Disallow /api/
Disallow /*/api/
Disallow /fld/
Disallow /*/fld/
Disallow /integrations/
Disallow /*/integrations/
Disallow /tool/
Disallow /*/tool/
Disallow /*collection/
Disallow /*swap_list/
Disallow /*wish_list/
Disallow /*ignore/
Disallow /*buy_list/
Disallow /*sell_list/
Disallow /*custom_list
Allow /*by_collection/
Allow /*by_swap_list/
Allow /*by_wish_list/

adsbot-google

Rule Path
Disallow *..
Disallow /teletalk/
Disallow /downloads/
Disallow /forums/download/
Disallow /main/set_language/
Disallow /gu
Disallow /ht
Disallow /kk
Disallow /ml
Disallow /mk
Disallow /pa/
Disallow /sw
Disallow /ta
Disallow /te
Allow /images
Disallow /*/self$
Disallow /*/self/
Disallow /*/new/
Disallow /*/online/
Disallow /*/edit/
Disallow /*/login$
Disallow /*/account$
Disallow /*/account/
Disallow /*/collectors/rate/
Disallow /*/collectors/log/
Disallow /*/collectors/list/friends/
Disallow /*/collectors/list/marked_as_friend/
Disallow /*/collectors/list/watchlist/
Disallow /*/collectors/list/best_matches/
Disallow /*/collectors/log_global/
Disallow /*/collectors/mark/
Disallow /*/collectors/my_profile
Disallow /*/collectors/edit_log
Disallow /*/collectors/translation_log
Disallow /*/directory/
Disallow /*/item/view_collectors/
Disallow /*/main/
Disallow /*/collectors/*/active/
Disallow /*/cart/
Disallow /*/seller/
Disallow /*/transaction/
Disallow /api/
Disallow /*/api/
Disallow /fld/
Disallow /*/fld/
Disallow /integrations/
Disallow /*/integrations/
Disallow /tool/
Disallow /*/tool/
Disallow /*collection/
Disallow /*swap_list/
Disallow /*wish_list/
Disallow /*ignore/
Disallow /*buy_list/
Disallow /*sell_list/
Disallow /*custom_list
Allow /*by_collection/
Allow /*by_swap_list/
Allow /*by_wish_list/

baiduspider

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

bubing

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

brandverity/1.0

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

daum

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

fast

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

java/1.6.0_10

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

yyspider

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

toscrawler

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

wesee

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

camontspider

Rule Path
Disallow /

spbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

xenu’s

Rule Path
Disallow /

xenu’s link sleuth 1.1c

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

archive.org bot

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

gigablast spider

Rule Path
Disallow /

ia_archiver-web.archive.org

Rule Path
Disallow /

picscout

Rule Path
Disallow /

tineye

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

ezooms robot

Rule Path
Disallow /

netestate ne crawler (+http://www.website-datenbank.de/)

Rule Path
Disallow /

wiseguys robot

Rule Path
Disallow /

turnitin robot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

pimonster

Rule Path
Disallow /

pimonster

Rule Path
Disallow /

pi-monster

Rule Path
Disallow /

eccp/1.0 (search@eniro.com)

Rule Path
Disallow /

psbot

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

zbot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

simplepie

Rule Path
Disallow /

pixray-seeker

Rule Path
Disallow /

boardreader

Rule Path
Disallow /

quantify

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

cuam

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

applebot

Rule Path
Disallow /

linespider

Rule Path
Disallow /

hyscore

Rule Path
Disallow /

*

Rule Path
Disallow *..
Disallow /teletalk/
Disallow /downloads/
Disallow /forums/download/
Disallow /main/set_language/
Disallow /gu
Disallow /ht
Disallow /kk
Disallow /ml
Disallow /mk
Disallow /pa/
Disallow /sw
Disallow /ta
Disallow /te
Allow /images
Disallow /*/self$
Disallow /*/self/
Disallow /*/new/
Disallow /*/online/
Disallow /*/edit/
Disallow /*/login$
Disallow /*/account$
Disallow /*/account/
Disallow /*/collectors/rate/
Disallow /*/collectors/log/
Disallow /*/collectors/list/friends/
Disallow /*/collectors/list/marked_as_friend/
Disallow /*/collectors/list/watchlist/
Disallow /*/collectors/list/best_matches/
Disallow /*/collectors/log_global/
Disallow /*/collectors/mark/
Disallow /*/collectors/my_profile
Disallow /*/collectors/edit_log
Disallow /*/collectors/translation_log
Disallow /*/directory/
Disallow /*/item/view_collectors/
Disallow /*/main/
Disallow /*/collectors/*/active/
Disallow /*/cart/
Disallow /*/seller/
Disallow /*/transaction/
Disallow /api/
Disallow /*/api/
Disallow /fld/
Disallow /*/fld/
Disallow /integrations/
Disallow /*/integrations/
Disallow /tool/
Disallow /*/tool/
Disallow /*unapproved/
Disallow /*collection/
Disallow /*swap_list/
Disallow /*wish_list/
Disallow /*ignore/
Disallow /*buy_list/
Disallow /*sell_list/
Disallow /*custom_list
Allow /*by_collection/
Allow /*by_swap_list/
Allow /*by_wish_list/
Back to top

Other Records

Field Value
sitemap https://colnect.com/s/sitemap_en.xml.gz
sitemap https://colnect.com/s/sitemap_af.xml.gz
sitemap https://colnect.com/s/sitemap_sq.xml.gz
sitemap https://colnect.com/s/sitemap_ar.xml.gz
sitemap https://colnect.com/s/sitemap_az.xml.gz
sitemap https://colnect.com/s/sitemap_bn.xml.gz
sitemap https://colnect.com/s/sitemap_bg.xml.gz
sitemap https://colnect.com/s/sitemap_be.xml.gz
sitemap https://colnect.com/s/sitemap_ca.xml.gz
sitemap https://colnect.com/s/sitemap_hr.xml.gz
sitemap https://colnect.com/s/sitemap_cs.xml.gz
sitemap https://colnect.com/s/sitemap_da.xml.gz
sitemap https://colnect.com/s/sitemap_nl.xml.gz
sitemap https://colnect.com/s/sitemap_et.xml.gz
sitemap https://colnect.com/s/sitemap_fi.xml.gz
sitemap https://colnect.com/s/sitemap_fr.xml.gz
sitemap https://colnect.com/s/sitemap_fy.xml.gz
sitemap https://colnect.com/s/sitemap_ka.xml.gz
sitemap https://colnect.com/s/sitemap_de.xml.gz
sitemap https://colnect.com/s/sitemap_el.xml.gz
sitemap https://colnect.com/s/sitemap_he.xml.gz
sitemap https://colnect.com/s/sitemap_hi.xml.gz
sitemap https://colnect.com/s/sitemap_hu.xml.gz
sitemap https://colnect.com/s/sitemap_id.xml.gz
sitemap https://colnect.com/s/sitemap_it.xml.gz
sitemap https://colnect.com/s/sitemap_ja.xml.gz
sitemap https://colnect.com/s/sitemap_ko.xml.gz
sitemap https://colnect.com/s/sitemap_lv.xml.gz
sitemap https://colnect.com/s/sitemap_lt.xml.gz
sitemap https://colnect.com/s/sitemap_ms.xml.gz
sitemap https://colnect.com/s/sitemap_no.xml.gz
sitemap https://colnect.com/s/sitemap_fa.xml.gz
sitemap https://colnect.com/s/sitemap_pl.xml.gz
sitemap https://colnect.com/s/sitemap_pt.xml.gz
sitemap https://colnect.com/s/sitemap_ro.xml.gz
sitemap https://colnect.com/s/sitemap_ru.xml.gz
sitemap https://colnect.com/s/sitemap_sr.xml.gz
sitemap https://colnect.com/s/sitemap_si.xml.gz
sitemap https://colnect.com/s/sitemap_sk.xml.gz
sitemap https://colnect.com/s/sitemap_sl.xml.gz
sitemap https://colnect.com/s/sitemap_es.xml.gz
sitemap https://colnect.com/s/sitemap_sv.xml.gz
sitemap https://colnect.com/s/sitemap_tl.xml.gz
sitemap https://colnect.com/s/sitemap_th.xml.gz
sitemap https://colnect.com/s/sitemap_tr.xml.gz
sitemap https://colnect.com/s/sitemap_uk.xml.gz
sitemap https://colnect.com/s/sitemap_ur.xml.gz
sitemap https://colnect.com/s/sitemap_br.xml.gz
sitemap https://colnect.com/s/sitemap_zt.xml.gz
Back to top

Comments

  • robots.txt for Colnect Collectors Community - https://colnect.com
  • Colnect has a lot of pages so misbehaved spiders may be blocked!
  • Happy collecting :)
  • msnbot gone berzerk, hopefully temporary, commented out 27/5/2020
  • User-agent: msnbot
  • Disallow: /
  • User-agent: bingbot
  • Disallow: /
  • User-agent: BingPreview
  • Disallow: /
  • User-agent: AdIdxBot
  • Disallow: /
  • Allow Facebook to preview pages
  • Czech SeznamBot added to lower request-rate
  • Disallow: /hy
  • Disallow: /mn
  • Annoying AdWords bot doesn't obey general rules and makes bogus requests
  • Disallow: /hy
  • Disallow: /mn
  • Crawlers that are kind enough to obey, but which we'd rather not have
  • unless they're feeding search engines.
  • Seems legit but for 10 visits a month paying in thousands of daily requests is too much
  • Some bots are known to be trouble, particularly those designed to copy
  • entire sites. Please obey robots.txt.
  • Ask Fasterfox plugin not to prefetch
  • Makes a lot of wrong requests - ignores this directive so it's totally blocked
  • Makes a lot of wrong requests
  • More useless bots
  • More bad SEO bots - taken from https://www.shoutmeloud.com/what-is-robots-txt-file-and-how-to-optimize-for-wordpress-blogs.html
  • Block pricepi
  • Block Eniro
  • no traffic from this "search engine"
  • https://megaindex.com/crawler
  • All the rest
  • Disallow: /hy
  • Disallow: /mn
  • Don't block images access
  • BEGINNING AUTOMATED PART - generated 2023-11-15T12:55:54+00:00
  • This part for intelligent bots which accept wildcards
  • END AUTOMATED PART - generated 2023-11-15T12:55:54+00:00
Back to top

Warnings

  • 4 invalid lines.
  • `request-rate` is not a known field.
Back to top