Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Regarding the adding of additional sinks/sources for various DB's

Copy link to this message
Re: Regarding the adding of additional sinks/sources for various DB's
As someone who just developed a sink, let me add my two cents.

If the intention is to separate “core Flume” from second class citizens like myself ( ;-) ), a contrib module only makes sense if those contributors can manage fixes and commit to their modules themselves.  Waiting for core developers to apply changes to modules they don’t want to work on will just leave maintainers like myself annoyed at waiting and core contributors annoyed at having to do it.  I think you’d have to hand out commit abilities to several people for there to be smiles all round.

If you don’t want to or can’t do that (understandable), maybe just let everyone do their own module management on GitHub or whatever, and provide a page that links to “add on” modules.  (This is the approach Elasticsearch takes, I think.)

-- Jeremy
On Nov 28, 2013, at 11:06, Hari Shreedharan <[EMAIL PROTECTED]> wrote:

> Juhani and others,
> I agree that it does make sense to add a contrib module to flume where
> non-hadoopy stuff can go. I will start a discussion on this early next week.
> Hari
> On Thursday, November 28, 2013, Steve Morin wrote:
>> Israel,
>> I guess my questions is why the suggestion to use the elastic search
>> model, is there something you see that's not working?
>> -Steve
>> On Mon, Nov 25, 2013 at 5:34 PM, Israel Ekpo <[EMAIL PROTECTED]<javascript:;>>
>> wrote:
>>> I think we can take a page of out the ElasticSearch playbook.
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html
>>> I like the model they follow.
>>> The Flume architecture makes it easy for plugins at any layer (source,
>>> interceptor, sink etc)
>>> Contributors can host plugins on github and manage the documentation and
>>> maintenance of the plugin.
>>> Others can chip it when possible to improve or maintain the plugins.
>>> This will still allow new features to the project without necessarily
>>> meaning that Flume committers are on the hook for maintaining it.
>>> *Author and Instructor for the Upcoming Book and Lecture Series*
>>> *Massive Log Data Aggregation, Processing, Searching and Visualization
>> with
>>> Open Source Software*
>>> *http://massivelogdata.com <http://massivelogdata.com>*
>>> On Mon, Nov 25, 2013 at 3:14 AM, Juhani Connolly <
>>> [EMAIL PROTECTED] <javascript:;>> wrote:
>>>> Hey guys,
>>>> What I write here is all just my personal opinion and I'm writing in
>>> hopes
>>>> of starting a discussion and/or getting feedback. I know I've not been
>>> very
>>>> active on the project recently(due to other engagements) but do still
>>> want
>>>> it to succeed and hope to find more time for it eventually.
>>>> Right now I see new/active issues for the addition of Redis and Kafka
>>>> sinks, and while they're nice features, I'm personally concerned about
>>>> feature bloat of the project. There are dozens of interceptors, sinks
>> and
>>>> sources that can be thought of, but most of them are very specific to a
>>>> specific use-case.
>>>> Every time we add a new component we're also committing to maintaining
>> it
>>>> over future releases, even if the original contributor gets too busy
>> for
>>>> it. The more such components get added, the more we will get distracted
>>>> from improving core features and getting rid of issues affecting them.
>>>> For these reasons I generally haven't submitted components we developed
>>>> for internal use(because they are too specific to our use cases), just
>>>> passing back fixes that fix bugs or apply to the core project.
>>>> For these reasons I think we may want to consider either a) being more
>>>> selective regarding additional component submissions or b) make a
>> contrib
>>>> directory to the project which includes the components but doesn't
>>>> guarrantee ongoing maintenance or compatibility.
>>>> On the flip side of course, taking approaches like this may discourage