Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Can Flume handle +100k events per seccond?


Copy link to this message
-
Can Flume handle +100k events per seccond?
Hi, this is my first post here. But i play with flume for some time now.
My question is how well flume scale?
Can Flume ingest +100k events per seccond? Has anyone tried something like
this?

I created simple test and results are really slow.
I wrote simple app with rpc client with fallback using flume sdk which is
reading dummy log file.
In the end i have two flume agents which are writing to hdfs.
rollInterval = 60
And in hdfs i get files with ~12MB.

Do i need to use some complex topology with 3 tier?
How many flume agents should write to hdfs?

Best regards.