Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: FLUME-1782 Elastic Search sink does not use UTC to determine the correct index to write to


+
Edward Sargisson 2013-04-09, 17:59
+
Edward Sargisson 2013-04-09, 17:59
Copy link to this message
-
Re: Review Request: FLUME-1782 Elastic Search sink does not use UTC to determine the correct index to write to
Israel Ekpo 2013-04-11, 10:23

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10379/#review18997
-----------------------------------------------------------

flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java
<https://reviews.apache.org/r/10379/#comment39478>

    I think you should
    
    1. Add a null check for the event headers before attempting to retrieve the timestamp header.
    
    2. Provide a strategy for still returning an index using the current time in cases where no valid timestamp header in present in the event.
- Israel Ekpo
On April 9, 2013, 5:59 p.m., Edward Sargisson wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10379/
> -----------------------------------------------------------
>
> (Updated April 9, 2013, 5:59 p.m.)
>
>
> Review request for Flume and Hari Shreedharan.
>
>
> Description
> -------
>
> This change gets the timestamp from the event and uses it (in UTC) to determine the name of the index to write to. This is required to match the behaviour of Logstash so that Kibana can find the log events.
> The previous code would use the current time and would do it in the timezone of the Flume agent's host.
>
>
> This addresses bug FLUME-1782.
>     https://issues.apache.org/jira/browse/FLUME-1782
>
>
> Diffs
> -----
>
>   flume-ng-doc/sphinx/FlumeUserGuide.rst 693c0d7
>   flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 1b3db14
>   flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java 94b95b1
>
> Diff: https://reviews.apache.org/r/10379/diff/
>
>
> Testing
> -------
>
> All unit tests and integration tests pass. A snapshot using commit 5b9d31f1ad228 and the patch for flume-1972 has passed our internal integration tests using customisations.
>
>
> Thanks,
>
> Edward Sargisson
>
>

+
Edward Sargisson 2013-04-11, 18:58
+
Israel Ekpo 2013-04-11, 10:28
+
Edward Sargisson 2013-04-11, 18:58
+
Edward Sargisson 2013-04-11, 19:03
+
Israel Ekpo 2013-04-17, 23:57
+
Edward Sargisson 2013-04-18, 23:39
+
Mike Percy 2013-04-19, 00:15
+
Edward Sargisson 2013-04-19, 18:04
+
Edward Sargisson 2013-04-22, 20:47