Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Regarding the adding of additional sinks/sources for various DB's


Copy link to this message
-
Re: Regarding the adding of additional sinks/sources for various DB's
Steve Morin 2013-11-29, 22:12
Frank,
  I would definitely vote for Kafka source/sink inclusion into flume.
-Steve
On Fri, Nov 29, 2013 at 7:11 AM, Frank Yao <[EMAIL PROTECTED]> wrote:

> I'm the 'guys' who is working on Kafka source/sinks:).
>
> A feature for popular, fast-growing and mature products is necessary to
> merge into Flume. Why?
> a) sources/sinks of mature products are really a motivation to flume users.
> b) developers have willing to add new sources/sinks to flume.
>
> For a), early there is a developer said, ' Having to shop around for
> various sources/sinks is more troublesome since I have to first find which
> flavor of a given sink is being maintained today, deal with licenses,
> incompatibilities, mismatch versions, upgrades, deployment, not fixed bugs
> and wondering if this is even going to work at all.' I thinks this is
> common for people who want to use flume but cannot found what he wanted at
> first.
> For b), if developers give plugins to flume and are rejected only because
> of keeping Flume lean, developers will lose their passion of contributing
> to Flume.
>
> I think if developers made sources/sinks for Flume, and he/she thought the
> sources/sinks were in great need by Flume users, he/she need to request a
> vote by committers, if most committers think it's necessary, then merge
> these to Flume. If not, then move these to  Flume-contrib projects like
> what ElasticSearch does.
>
>
>
> -----------------
>
> 姚仁捷 Frank Yao
> @超大杯摩卡星冰乐 <http://weibo.com/frankymryao>
> http://baniu.me
> Vipshop, Shanghai
>
>
>
>
> On Fri, Nov 29, 2013 at 3:17 PM, Otis Gospodnetic <
> [EMAIL PROTECTED]> wrote:
>
> > If you think "ecosystem", and an OSS project like Flume should very much
> > think ecosystem, then leaving things on Github, etc. probably makes more
> > sense.  Over the years (now over a decade!) I've witnessed what happens
> > with contrib/-type approach - authors need to have access to maintain
> their
> > stuff, make it work with the build system changes, make it work before
> > released, etc. etc,, which is all hard, and very often you can't just
> give
> > contrib authors Apache commit rights.  So instead of trying to pull
> > everything in, one should focus on *making developer-friendly core/APIs".
> >  Developers will then build tools that work with this core and naturally
> > create a rich ecosystem of tools around it.
> >
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> >
> >
> > On Nov 29, 2013 12:14 AM, "Jeremy Karlson" <[EMAIL PROTECTED]>
> > wrote:
> >
> > > As someone who just developed a sink, let me add my two cents.
> > >
> > > If the intention is to separate “core Flume” from second class citizens
> > > like myself ( ;-) ), a contrib module only makes sense if those
> > > contributors can manage fixes and commit to their modules themselves.
> > >  Waiting for core developers to apply changes to modules they don’t
> want
> > to
> > > work on will just leave maintainers like myself annoyed at waiting and
> > core
> > > contributors annoyed at having to do it.  I think you’d have to hand
> out
> > > commit abilities to several people for there to be smiles all round.
> > >
> > > If you don’t want to or can’t do that (understandable), maybe just let
> > > everyone do their own module management on GitHub or whatever, and
> > provide
> > > a page that links to “add on” modules.  (This is the approach
> > Elasticsearch
> > > takes, I think.)
> > >
> > > -- Jeremy
> > >
> > >
> > > On Nov 28, 2013, at 11:06, Hari Shreedharan <[EMAIL PROTECTED]
> >
> > > wrote:
> > >
> > > > Juhani and others,
> > > >
> > > > I agree that it does make sense to add a contrib module to flume
> where
> > > > non-hadoopy stuff can go. I will start a discussion on this early
> next
> > > week.
> > > >
> > > > Hari
> > > >
> > > > On Thursday, November 28, 2013, Steve Morin wrote:
> > > >
> > > >> Israel,
> > > >> I guess my questions is why the suggestion to use the elastic search