Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> multi-threaded elasticsearch sink


+
Allan Feid 2013-06-19, 15:00
+
Roshan Naik 2013-06-19, 18:17
Copy link to this message
-
Re: multi-threaded elasticsearch sink
Technically, even the HDFS sink uses only one thread to write to HDFS. The Async Hbase Sink writes using multiple threads (though they are hidden away from the sink itself - it is in the underlying API).  
Cheers,
Hari
On Wednesday, June 19, 2013 at 11:17 AM, Roshan Naik wrote:

> take a look at hdfs sink.
> -roshan
>
>
>
> On Wed, Jun 19, 2013 at 8:00 AM, Allan Feid <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > I'm not that great at Java at the moment, but it appears that the single threaded nature of the elasticsearch sink has trouble keeping up with ~5k events/second at 2k batch size. It looks like the HDFS sink has the ability to run multiple threads that write to the HDFS. I can get some performance increase by adding multiple ElasticSearch sinks to simulate parallelism, but it would be great for the sink itself to support multiple threads.
> >
> > Is there a sink example that should be used as a guide towards getting the same features in the elasticsearch sink?
> >
> > Thanks,
> > Allan
> >
> >
>
>
>

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB