Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: FLUME-1782 Elastic Search sink does not use UTC to determine the correct index to write to


+
Edward Sargisson 2013-04-09, 17:59
+
Edward Sargisson 2013-04-09, 17:59
+
Israel Ekpo 2013-04-11, 10:23
+
Edward Sargisson 2013-04-11, 18:58
+
Israel Ekpo 2013-04-11, 10:28
+
Edward Sargisson 2013-04-11, 18:58
+
Edward Sargisson 2013-04-11, 19:03
+
Israel Ekpo 2013-04-17, 23:57
Copy link to this message
-
Re: Review Request: FLUME-1782 Elastic Search sink does not use UTC to determine the correct index to write to
Edward Sargisson 2013-04-18, 23:39

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10379/
-----------------------------------------------------------

(Updated April 18, 2013, 11:39 p.m.)
Review request for Flume and Hari Shreedharan.
Changes
-------

Rework from Mike Percy's review:
* Event is no longer modified to carry the generated timestamp. This is a BREAKING change to the interface.
* Now use constants for timestamp values in the tests.
* Now use Charsets.UTF_8.
Description
-------

This change gets the timestamp from the event and uses it (in UTC) to determine the name of the index to write to. This is required to match the behaviour of Logstash so that Kibana can find the log events.
The previous code would use the current time and would do it in the timezone of the Flume agent's host.
This addresses bug FLUME-1782.
    https://issues.apache.org/jira/browse/FLUME-1782
Diffs (updated)
-----

  flume-ng-doc/sphinx/FlumeUserGuide.rst 38f2205
  flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchDynamicSerializer.java aa7ad39
  flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchEventSerializer.java dc6a093
  flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchLogStashEventSerializer.java 3638368
  flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 1b3db14
  flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchDynamicSerializer.java 43a4b12
  flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchLogStashEventSerializer.java 9dff4b0
  flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java 94b95b1

Diff: https://reviews.apache.org/r/10379/diff/
Testing
-------

All unit tests and integration tests pass. A snapshot using commit 5b9d31f1ad228 and the patch for flume-1972 has passed our internal integration tests using customisations.
Thanks,

Edward Sargisson

+
Mike Percy 2013-04-19, 00:15
+
Edward Sargisson 2013-04-19, 18:04
+
Edward Sargisson 2013-04-22, 20:47