Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - HDFS Sink Memory Leak


Copy link to this message
-
Re: HDFS Sink Memory Leak
Hari Shreedharan 2013-11-11, 16:03
This is because like you said you have too many files open at the same
time. HDFS stream classes keep a pretty large buffer (this is HDFS client
code not Flume) which will be cleaned up when the file is closed. Meeting
maxOpenFiles to a smaller number is a good way to handle this.

On Monday, November 11, 2013, David Sinclair wrote:

> I forgot to mention that map is contained in the HDFSEventSink class.
>
> Devin,
>
> Are you setting a roll interval? I use roll intervals so the .tmp files
> were getting closed, even if they were idle. They were just never being
> removed from that hashmap.
>
>
> On Mon, Nov 11, 2013 at 10:10 AM, DSuiter RDX <[EMAIL PROTECTED]<javascript:_e({}, 'cvml', '[EMAIL PROTECTED]');>
> > wrote:
>
>> David,
>>
>> This is insightful - I found the need to place an idleTimeout value in
>> the Flume config, but we were not running out of memory, we just found out
>> that lots of unclosed .tmp files got left laying around when the roll
>> occurred. I believe these are registering as under-replicated blocks as
>> well - in my pseudo-distributed testbed, I have 5 under-replicated
>> blocks...when the replication factor for pseudo-mode is "1" - and so we
>> don't like them in the actual cluster.
>>
>> Can you tell me, in your research, have you found a good way to close the
>> .tmp files out so they are properly acknowledged by HDFS/BucketWriter? Or
>> is simply renaming them sufficient? I've been concerned that the manual
>> rename approach might leave some floating metadata around, which is not
>> ideal.
>>
>> If you're not sure, don't sweat it, obviously. I was just wondering if
>> you already knew and could save me some empirical research time...
>>
>> Thanks!
>> *Devin Suiter*
>> Jr. Data Solutions Software Engineer
>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>> Google Voice: 412-256-8556 | www.rdx.com
>>
>>
>> On Mon, Nov 11, 2013 at 10:01 AM, David Sinclair <
>> [EMAIL PROTECTED] <javascript:_e({}, 'cvml',
>> '[EMAIL PROTECTED]');>> wrote:
>>
>>> Hi all,
>>>
>>> I have been investigating an OutOfMemory error when using the HDFS event
>>> sink. I have determined the problem to be with the
>>>
>>> WriterLinkedHashMap sfWriters;
>>>
>>> Depending on how you generate your file name/directory path, you can run
>>> out of memory pretty quickly. You need to either set the *idleTimeout*to some non-zero value or set the number of
>>> *maxOpenFiles*.
>>>
>>> The map keeps references to BucketWriter around longer than they are
>>> needed. I was able to reproduce this consistently and took a heap dump to
>>> verify that objects being kept around.
>>>
>>> I will update this Jira to reflect my findings
>>>
>>>
>>> https://issues.apache.org/jira/browse/FLUME-1326?jql=project%20%3D%20FLUME%20AND%20text%20~%20%22memory%20leak%22
>>>
>>> dave
>>>
>>
>>
>