Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Regarding the adding of additional sinks/sources for various DB's


Copy link to this message
-
Re: Regarding the adding of additional sinks/sources for various DB's
Frank,
  I would definitely vote for Kafka source/sink inclusion into flume.
-Steve
On Fri, Nov 29, 2013 at 7:11 AM, Frank Yao <[EMAIL PROTECTED]> wrote:

> I'm the 'guys' who is working on Kafka source/sinks:).
>
> A feature for popular, fast-growing and mature products is necessary to
> merge into Flume. Why?
> a) sources/sinks of mature products are really a motivation to flume users.
> b) developers have willing to add new sources/sinks to flume.
>
> For a), early there is a developer said, ' Having to shop around for
> various sources/sinks is more troublesome since I have to first find which
> flavor of a given sink is being maintained today, deal with licenses,
> incompatibilities, mismatch versions, upgrades, deployment, not fixed bugs
> and wondering if this is even going to work at all.' I thinks this is
> common for people who want to use flume but cannot found what he wanted at
> first.
> For b), if developers give plugins to flume and are rejected only because
> of keeping Flume lean, developers will lose their passion of contributing
> to Flume.
>
> I think if developers made sources/sinks for Flume, and he/she thought the
> sources/sinks were in great need by Flume users, he/she need to request a
> vote by committers, if most committers think it's necessary, then merge
> these to Flume. If not, then move these to  Flume-contrib projects like
> what ElasticSearch does.
>
>
>
> -----------------
>
> 姚仁捷 Frank Yao
> @超大杯摩卡星冰乐 <http://weibo.com/frankymryao>
> http://baniu.me
> Vipshop, Shanghai
>
>
>
>
> On Fri, Nov 29, 2013 at 3:17 PM, Otis Gospodnetic <
> [EMAIL PROTECTED]> wrote:
>
> > If you think "ecosystem", and an OSS project like Flume should very much
> > think ecosystem, then leaving things on Github, etc. probably makes more
> > sense.  Over the years (now over a decade!) I've witnessed what happens
> > with contrib/-type approach - authors need to have access to maintain
> their
> > stuff, make it work with the build system changes, make it work before
> > released, etc. etc,, which is all hard, and very often you can't just
> give
> > contrib authors Apache commit rights.  So instead of trying to pull
> > everything in, one should focus on *making developer-friendly core/APIs".
> >  Developers will then build tools that work with this core and naturally
> > create a rich ecosystem of tools around it.
> >
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> >
> >
> > On Nov 29, 2013 12:14 AM, "Jeremy Karlson" <[EMAIL PROTECTED]>
> > wrote:
> >
> > > As someone who just developed a sink, let me add my two cents.
> > >
> > > If the intention is to separate “core Flume” from second class citizens
> > > like myself ( ;-) ), a contrib module only makes sense if those
> > > contributors can manage fixes and commit to their modules themselves.
> > >  Waiting for core developers to apply changes to modules they don’t
> want
> > to
> > > work on will just leave maintainers like myself annoyed at waiting and
> > core
> > > contributors annoyed at having to do it.  I think you’d have to hand
> out
> > > commit abilities to several people for there to be smiles all round.
> > >
> > > If you don’t want to or can’t do that (understandable), maybe just let
> > > everyone do their own module management on GitHub or whatever, and
> > provide
> > > a page that links to “add on” modules.  (This is the approach
> > Elasticsearch
> > > takes, I think.)
> > >
> > > -- Jeremy
> > >
> > >
> > > On Nov 28, 2013, at 11:06, Hari Shreedharan <[EMAIL PROTECTED]
> >
> > > wrote:
> > >
> > > > Juhani and others,
> > > >
> > > > I agree that it does make sense to add a contrib module to flume
> where
> > > > non-hadoopy stuff can go. I will start a discussion on this early
> next
> > > week.
> > > >
> > > > Hari
> > > >
> > > > On Thursday, November 28, 2013, Steve Morin wrote:
> > > >
> > > >> Israel,
> > > >> I guess my questions is why the suggestion to use the elastic search
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB