Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - [DISCUSS] Flume 1.4 release plan


Copy link to this message
-
Re: [DISCUSS] Flume 1.4 release plan
Edward Sargisson 2013-06-21, 21:59
Hi all,
I thought I'd give an update from the ElasticSearchSink perspective. I know
that we're very keen to see a 1.4 release and will test RCs, etc. when they
come out when priorities allow.

1. The patch to build with elasticsearch 0.90.1 is in FLUME-2049 and the
review is here: https://reviews.apache.org/r/11581/
I think we really need that one in Flume because otherwise anybody who
wants to use Flume with anything other than elasticsearch 0.19 (really old)
will have to do a custom compile.

2. I mentioned issues with Flume logging via itself. I got those to a point
where I thought I understood the problem and have raised work items either
in Flume or with Log4j2 to document. These do not hold up Flume 1.4.0

3. Ideally, we'd do FLUME-2092 to document that the index name needs to
match a Kibana setting.

4. It would probably be nice to get the patch from FLUME-2089 in
(unexpected character sets cause a YAMLException). However, I think we need
a unit test for this.

Cheers,
Edward

On Wed, May 22, 2013 at 9:28 AM, Edward Sargisson <[EMAIL PROTECTED]> wrote:

> Hi All,
> +1/+1 for 1.4 and Mike.
>
>
> I'm very keen to have a 1.4 for the environments I manage. There's a lot
> of stuff I'm keen on in there.
>
> On my pre-1.4 list:
> 1. compile with elasticsearch 0.90
> 2. figure out file channel state issue which is stopping Flume logging via
> itself.
>
> 1. Currently we compile with es 0.19. If somebody wants to run es 0.20
> they have to recompile (es made an interface change that is source
> compatible but requires a recompile). es 0.90 has been out for 2-ish weeks
> so safe enough to change the compile to. I think I'll raise an empty Jira
> to record this.
>
> 2. I haven't reported this because I haven't isolated it well enough. I'm
> having issues with the 1.3.1 file channel which I'd like to resolve.
>
> Cheers,
> Edward
>
> "Hi folks,
> We have had over 100 commits since 1.3.1, and a bunch of new features and
> improvements including a Thrift source, much improved ElasticSearch sink,
> support for a new plugins directory and layout, compression support in the
> avro sink/source, improved checkpointing in the file channel and more, plus
> a lot of bug fixes.
>
> It seems to me that it's time to start thinking about cutting a 1.4
> release. I would be happy to volunteer to RM the release. Worth noting that
> I will be unavailable for the next two weeks... but after that I'd be happy
> to pick this up and run with it. That's also a decent amount of time for
> people  to get moving on patches and reviews for their favorite features,
> bug fixes, etc.
>
> If this all sounds OK, I'd like to suggest targeting the last week of June
> as a release date. If we can release in time for Hadoop Summit then that
> would be pretty nice. Otherwise, if something comes up and we can't get the
> release out that week, let's shoot for the first week of July at the
> latest.
>
> Please let me know your thoughts.
>
> Regards,
> Mike
>
> +1 for Flume 1.4
> +1 for Mike being RM.
>
>
> Cheers,
> Hari"
>