Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> How to use Flume Client perl API to send data to ElasticSearch?


Copy link to this message
-
How to use Flume Client perl API to send data to ElasticSearch?
Hi Shushuai,
The Flume event must have a header with the name timestamp (all lower case)
which is Unix milliseconds since 1970.

Without this, the event gets sent to elasticsearch but Kibana can never
read it.

The ElasticSearchSink converts the timestamp (millis) header into the
@timestamp for Kibana.

Cheers,
Edward

"
Hi,

I am sending data in JSON format using Flume Client perl API to Flume Avro
source then ElasticSearchSink to an ElasticSearch engine. I could send the
data to ElasticSearch engine since I saw new indexes created in the engine,
but the data are not formatted correctly or partially lost since they could
not be viewed via Kibana.

An example of the data in JSON format is:

{"@timestamp":"2013-05-28T23:59:43.300Z","@source_host":"Source host
name","@type":"flume-input","@fields":{"FIELD_PRIORITY":"DEBUG","FIELD_CLASS":"procedure.engine","FIELD_FILE":"/scratch/abc/abc.log","FIELD_MESSAGE":"The
debug message","FIELD_TIME":"1369785558004"}}

Earlier, I used LogStash to pull JSON data from Redis to ElasticSearch, and
the LogStash conf file looks like:

input {
    redis {
        host => "abc"
        type => "redis-input"
        data_type => "list"
        key => "logstash"
        format => "json"
    }
}
output {
    elasticsearch {
        embedded => false
        bind_host => "abc"
        host => "abc"
        port => 9300
  }
}

The data were correctly input to ElasticSearch and could be viewed via
Kibana.

The sample perl API call for FlumeNG shown in Flume Client doc is:

use Flume::Client;
my $ng_client = Flume::Client::Transceiver::Socket->new(host => 'host
name', port => 41414);
my $ng_requestor = Flume::Client::Requestor::FlumeNG->new(client =>
$ng_client);
 my ($result, $response) = $ng_requestor->request('appendBatch', [{ headers
=> {}, body => "hello, this is sent from perl (using FlumeNG)"}]);
print "$response\n";    # response will be 'OK' on success

I tried putting the JSON data in "body" or "headers" or some other ways in
the above example, but could not get the right results in ElasticSearch.
Could someone tell me how to use the API to send the above example data to
ElasticSearch?

Shushuai"
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB