Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> multi-threaded elasticsearch sink


+
Allan Feid 2013-06-19, 15:00
Copy link to this message
-
Re: multi-threaded elasticsearch sink
take a look at hdfs sink.
-roshan
On Wed, Jun 19, 2013 at 8:00 AM, Allan Feid <[EMAIL PROTECTED]> wrote:

> I'm not that great at Java at the moment, but it appears that the single
> threaded nature of the elasticsearch sink has trouble keeping up with ~5k
> events/second at 2k batch size. It looks like the HDFS sink has the ability
> to run multiple threads that write to the HDFS. I can get some performance
> increase by adding multiple ElasticSearch sinks to simulate parallelism,
> but it would be great for the sink itself to support multiple threads.
>
> Is there a sink example that should be used as a guide towards getting the
> same features in the elasticsearch sink?
>
> Thanks,
> Allan
>
+
Hari Shreedharan 2013-06-19, 18:30
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB