Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> multi-threaded elasticsearch sink


Copy link to this message
-
multi-threaded elasticsearch sink
I'm not that great at Java at the moment, but it appears that the single
threaded nature of the elasticsearch sink has trouble keeping up with ~5k
events/second at 2k batch size. It looks like the HDFS sink has the ability
to run multiple threads that write to the HDFS. I can get some performance
increase by adding multiple ElasticSearch sinks to simulate parallelism,
but it would be great for the sink itself to support multiple threads.

Is there a sink example that should be used as a guide towards getting the
same features in the elasticsearch sink?

Thanks,
Allan
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB