Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Does Flume need a workload generator?


Copy link to this message
-
Re: Does Flume need a workload generator?
Mike Percy 2012-07-17, 04:41
Patrick,
Will and I have put together a couple of classes that do much (not all) of
what you're describing for a Syslog TCP client and an Avro RPC client. I've
been meaning to commit them to Flume SVN but I haven't gotten around to it
yet. Here is the syslog one:
https://github.com/mpercy/flume-load-gen/blob/master/src/main/java/org/apache/flume/tcphammer/Hammer.java

I can give you an Avro RPC one which is similar.

Regards,
Mike

On Mon, Jul 16, 2012 at 9:28 PM, Patrick Wendell <[EMAIL PROTECTED]> wrote:

> Hi All,
>
> I was trying to debug a memory leak and it doesn't seem like there is
> a good way to generate workload for Flume at a configurable rate, or
> with configurable characteristics like payload size and header size.
> The sequence source is something like this, but you can't control the
> rate of event generation at all.
>
> What do people do now when benchmarking or debugging Flume? I was
> thinking of writing a source that generates event streams with
> configurable properties. Would that be useful? Or is this need already
> met somehow else?
>
> - Patrick
>