Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Regarding the adding of additional sinks/sources for various DB's


Copy link to this message
-
Re: Regarding the adding of additional sinks/sources for various DB's
On 11/29/2013 08:17 AM, Otis Gospodnetic wrote:
> If I understood this correctly, then I think you can also think of this
> from the opposite direction.  Instead of moving sources/sinks OUT of Flume,
> let them develop outside Flume, and if they prove to be good, in high
> demand, often used with Flume, or in need of "adoption" by core Flume
> developers, then BRING IN such sources/sinks and, optionally(?), their
> authors with them.  Maybe this way you can have the best of both worlds.
>
> If I develop source/sink for the new, fancy, FooBarBaz data store, can I
> get it in Flume?  No.  and that's good, because if FooBarBaz doesn't take
> off, who'll want to maintain it?
> If I develop source/sink for Kafka, which is known to be very popular and
> often used with Flume, and there are either core Flume developers
> interested in maintaining this source/sink, or if there are external
> developers who have been working on this outside Apache (i.e. the chances
> are they'll follow the project to Apache and continue contributing to it,
> eventually earning the committer rights), then this source/sink for Kafka
> should be considered for adoption by Flume.
>
> This way you let the external ecosystem live on, develop independently,
> compete for user and developer adoption, and once good sources/sinks emerge
> from there, they can be moved under Flume if everyone involved agrees
> that's the right thing to do.
>
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
If a component gets developed outside of Apache, what is the cost to
integrate it? How would that work? I assume one would have to go through
a code grant. If so, would a developer have to go track all past
contributors to agree on the code grant terms? Or should a CLA be
prepared in case of such event?
Why is it bad to get a new source/sink for the new, fancy, FooBrBaz data
store? What if it does take off? I would contend that the success of
that data store does not matter as long as there is a community
interested in it.
And why is it so costly to include it when it is so cheap to remove it
when no one cares about it anymore?
Including people from the outer layers also increases the chances of
getting them interested in the core of Apache Flume. They will most
likely notice something to improve somewhere else and start helping in
core parts. Even more so if they are going through all the steps to get
their code in Apache Flume.
Imho, the cost on passing on possible contributors is quite high
comparing to deleting unmaintained parts. Not just for the core Apache
Flume developers, but for the users in general as well.
Thanks,
Bruno