Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - multi-threaded elasticsearch sink


+
Allan Feid 2013-06-19, 15:00
Copy link to this message
-
Re: multi-threaded elasticsearch sink
Roshan Naik 2013-06-19, 18:17
take a look at hdfs sink.
-roshan
On Wed, Jun 19, 2013 at 8:00 AM, Allan Feid <[EMAIL PROTECTED]> wrote:

> I'm not that great at Java at the moment, but it appears that the single
> threaded nature of the elasticsearch sink has trouble keeping up with ~5k
> events/second at 2k batch size. It looks like the HDFS sink has the ability
> to run multiple threads that write to the HDFS. I can get some performance
> increase by adding multiple ElasticSearch sinks to simulate parallelism,
> but it would be great for the sink itself to support multiple threads.
>
> Is there a sink example that should be used as a guide towards getting the
> same features in the elasticsearch sink?
>
> Thanks,
> Allan
>
+
Hari Shreedharan 2013-06-19, 18:30