Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> How to use Flume Client perl API to send data to ElasticSearch?

shushuai zhu 2013-06-13, 03:47
shushuai zhu 2013-06-14, 03:03
Copy link to this message
How to use Flume Client perl API to send data to ElasticSearch?
Hi Shushuai,
The Flume event must have a header with the name timestamp (all lower case)
which is Unix milliseconds since 1970.

Without this, the event gets sent to elasticsearch but Kibana can never
read it.

The ElasticSearchSink converts the timestamp (millis) header into the
@timestamp for Kibana.



I am sending data in JSON format using Flume Client perl API to Flume Avro
source then ElasticSearchSink to an ElasticSearch engine. I could send the
data to ElasticSearch engine since I saw new indexes created in the engine,
but the data are not formatted correctly or partially lost since they could
not be viewed via Kibana.

An example of the data in JSON format is:

{"@timestamp":"2013-05-28T23:59:43.300Z","@source_host":"Source host
debug message","FIELD_TIME":"1369785558004"}}

Earlier, I used LogStash to pull JSON data from Redis to ElasticSearch, and
the LogStash conf file looks like:

input {
    redis {
        host => "abc"
        type => "redis-input"
        data_type => "list"
        key => "logstash"
        format => "json"
output {
    elasticsearch {
        embedded => false
        bind_host => "abc"
        host => "abc"
        port => 9300

The data were correctly input to ElasticSearch and could be viewed via

The sample perl API call for FlumeNG shown in Flume Client doc is:

use Flume::Client;
my $ng_client = Flume::Client::Transceiver::Socket->new(host => 'host
name', port => 41414);
my $ng_requestor = Flume::Client::Requestor::FlumeNG->new(client =>
 my ($result, $response) = $ng_requestor->request('appendBatch', [{ headers
=> {}, body => "hello, this is sent from perl (using FlumeNG)"}]);
print "$response\n";    # response will be 'OK' on success

I tried putting the JSON data in "body" or "headers" or some other ways in
the above example, but could not get the right results in ElasticSearch.
Could someone tell me how to use the API to send the above example data to

shushuai zhu 2013-06-14, 18:52
Edward Sargisson 2013-06-18, 21:48
shushuai zhu 2013-06-19, 04:14