Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume 1.3.0 - NFS + File Channel Performance


Copy link to this message
-
Re: Flume 1.3.0 - NFS + File Channel Performance
Hi,

Yes it'd be great to get 1763 and 1794 into 1.3.1. I don't have time
right at this point in time. If another committer does, I'd love to
vote on an RC! :)

Brock

On Wed, Dec 19, 2012 at 4:34 AM, Rakos, Rudolf
<[EMAIL PROTECTED]> wrote:
> Brock, Hari,
>
> I can confirm that the patch in FLUME-1794 fixes the performance issue.
>
> I was wondering whether it is possible to ask for a new release (1.3.1) including the recent File Channel bug fixes?
>
>   Trunk: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=history;f=flume-ng-channels/flume-file-channel;h=cc779e886b4d6290723a43b4f874239150d93475;hb=trunk
>   1.3.0: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=history;f=flume-ng-channels/flume-file-channel;h=cc93d99eac6d631e9200d122928d5e307621b4fe;hb=refs/heads/flume-1.3.0
>
> Unfortunately we cannot use trunk, and waiting for Flume 1.4.0 could take a few months.
> It's not a big problem if we need to stick with Flume 1.2.0, but according to Juhani Connolly this was causing high CPU usage with non-NFS File Channels too, so I think maybe it would be better for the community.
>
> Regards,
> Rudolf
>
> -----Original Message-----
> From: Rakos, Rudolf (ISGT)
> Sent: Wednesday, December 19, 2012 9:10 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Flume 1.3.0 - NFS + File Channel Performance
>
> Brock, Hari,
>
> Thank you very much for looking so quickly into this.
>
> We're aware that the general performance will not be that great using NFS, but having some "last minute" data on failover scenarios could be worth the performance cost.
>
> You were right.
> I've taken some thread dumps and I can confirm that FLUME-1609 (File.getUsableSpace calls) are causing the issue. (I just don't understand how could I miss this hot spot during profiling.)
>
> I'll check whether the patch in FLUME-1794 fixes this.
>
> Thanks,
> Rudolf
>
> -----Original Message-----
> From: Brock Noland [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, December 18, 2012 10:09 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Flume 1.3.0 - NFS + File Channel Performance
>
> Hi,
>
> If you do have a chance, it would great to hear if the patch attached to this JIRA (https://issues.apache.org/jira/browse/FLUME-1794) fixes the performance problem.
>
> Brock
>
> On Tue, Dec 18, 2012 at 11:25 AM, Brock Noland <[EMAIL PROTECTED]> wrote:
>> Yeah I think we should do that check in the background and then update
>> a flag. This how hdfs and mapred do it.
>>
>> On Tue, Dec 18, 2012 at 11:04 AM, Hari Shreedharan
>> <[EMAIL PROTECTED]> wrote:
>>> Yep. The disk space calls require an NFS call for each write, and
>>> that slows things down a lot.
>>>
>>> --
>>> Hari Shreedharan
>>>
>>> On Tuesday, December 18, 2012 at 8:43 AM, Brock Noland wrote:
>>>
>>> We'd need those thread dumps to help confirm but I bet that
>>> FLUME-1609 results in a NFS call on each operation on the channel.
>>>
>>> If that is true, that would explain why it works well on local disk.
>>>
>>> Brock
>>>
>>> On Tue, Dec 18, 2012 at 10:17 AM, Brock Noland <[EMAIL PROTECTED]> wrote:
>>>
>>> Hi,
>>>
>>> Hmm, yes in general performance is not going to be great over NFS,
>>> but there haven't been any FC changes that stick out here.
>>>
>>> Could you take 10 thread dumps of the agent running the file channel
>>> and 10 thread dumps of the agent sending data to the agent with the
>>> file channel? (You can address them to myself directly since the list
>>> won't take attachements.)
>>>
>>> Are there any patterns, like it works for 40 seconds then times out
>>> and then works for 39 seconds, etc?
>>>
>>> Brock
>>>
>>> On Tue, Dec 18, 2012 at 10:07 AM, Rakos, Rudolf
>>> <[EMAIL PROTECTED]> wrote:
>>>
>>> Hi,
>>>
>>>
>>>
>>> We’ve run into a strange problem regarding NFS and File Channel
>>> performance while evaluating the new version of Flume.
>>>
>>> We had no issues with the previous version (1.2.0).
>>>
>>>
>>>
>>> Our configuration looks like this:
>>>
>>> · Node1:
>>> (Avro RPC Clients ->) Avro Source and Custom Sources -> File Channel

Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/