Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> FW: Flume Service Takes Lot of Memory


Copy link to this message
-
RE: Flume Service Takes Lot of Memory
Hi Alex,
The memory is getting used at flume agent. Setting the variables MASTER_WATCHDOG & MASTERI_WATCHDOG at flume master will work for agent memory as well? Or do we have some other option as flume agent?

Regards,
Deepak

-----Original Message-----
From: Alexander Alten-Lorenz [mailto:[EMAIL PROTECTED]]
Sent: Thursday, January 17, 2013 2:18 AM
To: [EMAIL PROTECTED]
Subject: Re: Flume Service Takes Lot of Memory

Hi,

To add JVM options in flume we have to edit the watchdog line in bin/flume into:
MASTER_WATCHDOG="java -XX:+HeapDumpOnOutOfMemoryError ${JOPTS} -Dpid=$$ -Dpidfile=$MASTERPID $WATCHDOG_CLASS"
MASTERI_WATCHDOG="java -XX:+HeapDumpOnOutOfMemoryError ${JOPTS} -Dpid=$$ -Dpidfile=$MASTERPID $WATCHDOG_CLASS"

and restart Flume master.

flume-env.sh will not be used, because bin/flume honors only the variables FLUME_PID_DIR and FLUME_IDENT_STRING

Ensure that you have provided the jvm with sufficient heap space. By default Flume starts the jvm with it's default heap allocation, which differs depending on the jvm version, the host type (os, 32/64 bit, etc...), total host memory available, as well as other issues.

The environment variable UOPTS can be used to pass additional jvm parameters when running Flume. e.g.

$ UOPTS="-Xms1g -Xmx2g" bin/flume node

which starts a flume node with an initial heap of one gig and a max heap of two gig. See "java -h" or "java -X" for more details on available jvm options.

cheers,
 Alex

On Jan 17, 2013, at 8:10 AM, "Kumar, Deepak8" <[EMAIL PROTECTED]> wrote:

> Hi Alex,
> Thanks for assisting. I do have the following settings in flume-env.sh
> file export UOPTS="-Xms256m -Xmx512m"
>
> But once I do top, it shows this much virtual memory
>
> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 8571 root      21   0 1209m 424m  11m S  2.0  5.3  53:05.95 java -Dflume.log.dir=/var/log/flume  4957 root  
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Alexander Alten-Lorenz [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, January 17, 2013 1:58 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Flume Service Takes Lot of Memory
>
> Hi Deepak,
>
> This is a normal behavior, you can tweak flume's (OG) settings by adding some Xmx params:
>
> UOPTS="-Xms1g -Xmx2g" bin/flume node
>
> Of course you can also tweak the bin/flume script.
>
> - Alex
>
> On Jan 17, 2013, at 7:34 AM, "Kumar, Deepak8" <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>> Could you please guide?
>>
>> Regards,
>> Deepak
>>
>> -----Original Message-----
>> From: Kumar, Deepak8 [CCC-OT_IT NE]
>> Sent: Wednesday, January 16, 2013 1:29 PM
>> To: [EMAIL PROTECTED]
>> Subject: Flume Service Takes Lot of Memory
>>
>> Hi,
>> I am using flume OG 0.9, the flume agent seems to take almost 1Gig Virtual Memory. Is it uses this much normally or do we have some configuration settings?  
>>
>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> 8571 root      21   0 1209m 424m  11m S  2.0  5.3  53:05.95 java -Dflume.log.dir=/var/log/flume  4957 root  
>>
>> Regards,
>> Deepak
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF