Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: FLUME-2015 ElasticSearchSink: need access to IndexRequestBuilder instance during flume event processing


+
Tim Bacon 2013-04-29, 08:43
+
Tim Bacon 2013-05-03, 00:24
Copy link to this message
-
Re: Review Request: FLUME-2015 ElasticSearchSink: need access to IndexRequestBuilder instance during flume event processing
Mike Percy 2013-05-08, 20:05

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10835/#review20339
-----------------------------------------------------------
Hi Tim, thanks for the patch!

If Edward is ok with these changes then I'm +1 on them, except for the API breakage below. We need to maintain backwards compatibility with the existing API implementation.
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchEventSerializer.java
<https://reviews.apache.org/r/10835/#comment41900>

    This is an API-breaking change for existing plugins.
    
    Instead of this, a new interface should be created if this change needs to be made, and reflection (instanceof) should be used at the configuration layer to decide which serializer interface was implemented so that a separate codepath can be used.
- Mike Percy
On May 3, 2013, 12:24 a.m., Tim Bacon wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10835/
> -----------------------------------------------------------
>
> (Updated May 3, 2013, 12:24 a.m.)
>
>
> Review request for Flume and Israel Ekpo.
>
>
> Description
> -------
>
> This change adds an ElasticSearchIndexRequestBuilderFactory interface to allow users of an ElasticSearchSink to have greater control over the actual indexing of events that arrive at the sink. It is intended to meet the needs of:
> - my own non-Kibana/non-logging-event use case
> - the specific "allow for indexing an id-field" needs of FLUME-1972
> - the (UTC) determination of the index to write to per FLUME-1782
>
> This patch is backwards-compatible and imposes no changes on existing users of the sink.
>
>
> This addresses bug FLUME-2015.
>     https://issues.apache.org/jira/browse/FLUME-2015
>
>
> Diffs
> -----
>
>   flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java PRE-CREATION
>   flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchEventSerializer.java dc6a093
>   flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchIndexRequestBuilderFactory.java PRE-CREATION
>   flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 1b3db14
>   flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/EventSerializerIndexRequestBuilderFactory.java PRE-CREATION
>   flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java 2edacdc
>   flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java PRE-CREATION
>   flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java 94b95b1
>
> Diff: https://reviews.apache.org/r/10835/diff/
>
>
> Testing
> -------
>
> Unit tests added and run.
> Patch applied and run successfully in my own flume test environment.
>
>
> Thanks,
>
> Tim Bacon
>
>

+
Mike Percy 2013-05-08, 20:29
+
Tim Bacon 2013-05-09, 09:38
+
Tim Bacon 2013-05-10, 10:08
+
Mike Percy 2013-05-10, 20:19