Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - multi-threaded elasticsearch sink


Copy link to this message
-
Re: multi-threaded elasticsearch sink
Hari Shreedharan 2013-06-19, 18:30
Technically, even the HDFS sink uses only one thread to write to HDFS. The Async Hbase Sink writes using multiple threads (though they are hidden away from the sink itself - it is in the underlying API).  
Cheers,
Hari
On Wednesday, June 19, 2013 at 11:17 AM, Roshan Naik wrote:

> take a look at hdfs sink.
> -roshan
>
>
>
> On Wed, Jun 19, 2013 at 8:00 AM, Allan Feid <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > I'm not that great at Java at the moment, but it appears that the single threaded nature of the elasticsearch sink has trouble keeping up with ~5k events/second at 2k batch size. It looks like the HDFS sink has the ability to run multiple threads that write to the HDFS. I can get some performance increase by adding multiple ElasticSearch sinks to simulate parallelism, but it would be great for the sink itself to support multiple threads.
> >
> > Is there a sink example that should be used as a guide towards getting the same features in the elasticsearch sink?
> >
> > Thanks,
> > Allan
> >
> >
>
>
>