Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Can Flume handle +100k events per seccond?


Copy link to this message
-
Can Flume handle +100k events per seccond?
Hi, this is my first post here. But i play with flume for some time now.
My question is how well flume scale?
Can Flume ingest +100k events per seccond? Has anyone tried something like
this?

I created simple test and results are really slow.
I wrote simple app with rpc client with fallback using flume sdk which is
reading dummy log file.
In the end i have two flume agents which are writing to hdfs.
rollInterval = 60
And in hdfs i get files with ~12MB.

Do i need to use some complex topology with 3 tier?
How many flume agents should write to hdfs?

Best regards.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB