Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Does Flume need a workload generator?


Copy link to this message
-
Re: Does Flume need a workload generator?
Patrick,
Will and I have put together a couple of classes that do much (not all) of
what you're describing for a Syslog TCP client and an Avro RPC client. I've
been meaning to commit them to Flume SVN but I haven't gotten around to it
yet. Here is the syslog one:
https://github.com/mpercy/flume-load-gen/blob/master/src/main/java/org/apache/flume/tcphammer/Hammer.java

I can give you an Avro RPC one which is similar.

Regards,
Mike

On Mon, Jul 16, 2012 at 9:28 PM, Patrick Wendell <[EMAIL PROTECTED]> wrote:

> Hi All,
>
> I was trying to debug a memory leak and it doesn't seem like there is
> a good way to generate workload for Flume at a configurable rate, or
> with configurable characteristics like payload size and header size.
> The sequence source is something like this, but you can't control the
> rate of event generation at all.
>
> What do people do now when benchmarking or debugging Flume? I was
> thinking of writing a source that generates event streams with
> configurable properties. Would that be useful? Or is this need already
> met somehow else?
>
> - Patrick
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB