Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for header substitution in indexName and indexType


Copy link to this message
-
Re: Review Request 20601: Flume-2273 ElasticSearchSink: Add handling for header substitution in indexName and indexType
Satoshi Iijima 2014-05-01, 10:23

This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20601/

(Updated May 1, 2014, 7:22 p.m.)
Review request for Flume.
Changes

minor fix in doc
Repository: flume-git
Description

The ElasticSearchSink would be improved by allowing for header substitution in the indexName and indexType property.
A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value.
The HDFS sink already supports header substitution so a similar format would maintain consistency.

Example is below:

a1.sinks.k1.type = elasticsearch
a1.sinks.k1.indexName = %{index-name}
a1.sinks.k1.indexType = %{index-type}
a1.sinks.k1.hostNames = 127.0.0.1:9200
a1.sinks.k1.channel = c1
Diffs (updated)

  flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
  flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java de84b95
  flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java 39b6db5
  flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java 19079af
  flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java a8603a4
  flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java ff95e30
  flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java e9ed0b4
  flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java 807a9c7

Diff: https://reviews.apache.org/r/20601/diff/
Testing
Thanks,

Satoshi Iijima