Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Does Flume need a workload generator?

Copy link to this message
Re: Does Flume need a workload generator?
Will and I have put together a couple of classes that do much (not all) of
what you're describing for a Syslog TCP client and an Avro RPC client. I've
been meaning to commit them to Flume SVN but I haven't gotten around to it
yet. Here is the syslog one:

I can give you an Avro RPC one which is similar.


On Mon, Jul 16, 2012 at 9:28 PM, Patrick Wendell <[EMAIL PROTECTED]> wrote:

> Hi All,
> I was trying to debug a memory leak and it doesn't seem like there is
> a good way to generate workload for Flume at a configurable rate, or
> with configurable characteristics like payload size and header size.
> The sequence source is something like this, but you can't control the
> rate of event generation at all.
> What do people do now when benchmarking or debugging Flume? I was
> thinking of writing a source that generates event streams with
> configurable properties. Would that be useful? Or is this need already
> met somehow else?
> - Patrick