Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Avro 2 Avro transfer on Flume 1.3 screamingly slow


Copy link to this message
-
Re: Avro 2 Avro transfer on Flume 1.3 screamingly slow
Hari Shreedharan 2013-11-11, 17:06
Are you sure it is not the spooling directory source that is not able to write more data. What makes you think that it is avro sink to source that tis the issue? It just might be that your source is not writing data fast enough or your channel is getting full too fast and the channel is backing off too often?
Thanks,
Hari
On Monday, November 11, 2013 at 7:16 AM, Christopher Shannon wrote:

> Hello,
>
> I am trying to get a decent throughput rate of transfer for an Avro sink to Avro Source transfer, but the best I've been able to get is about 300 records a minute. This thread (http://mail-archives.apache.org/mod_mbox/flume-user/201309.mbox/%[EMAIL PROTECTED]%3e) on the mailing list suggests that hardware and disk partitioning is the main consideration to achieving good throughput.
>
> My setup is simple:
>
> Agent 1:
> Source: SpoolingDirectory
> Channel: Memory
> Sink: Avro
>
> Agent 2:
> Source: Avro
> Channel: Memory
> Sink: HDFS
>
> So, what else needs to be done to optimize the data transfer?
>
> All the best,
>
> Chris
>
> p.s. Anticipating suggestion that I upgrade to Flume 1.4, I work for a big company that has acquired Flume bundled through another product from an even bigger company. They will not consider an upgrade at this time. (And given the very slow throughput I'm seeing, I would think that 1.3 can do a lot better.)
>
>
>