Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> flume use quite a lot of direct momery and blast


Copy link to this message
-
Re: flume use quite a lot of direct momery and blast
Are you using hdfs sink?  If so please share your configuration.

I have seen this when the sink keeps many open files...each file holding on
to a native compression buffer.
On Jan 22, 2014 1:37 AM, "Shangan Chen" <[EMAIL PROTECTED]> wrote:

> Thanks a lot for your reply. I deployed flume on ubuntu and what confused
> me is not about the virtual address space but NIO Direct Memory increasing
> boundlessly. It will finally cause the machine to report swap is low.
>
>
> On Wed, Jan 22, 2014 at 1:08 AM, Brock Noland <[EMAIL PROTECTED]> wrote:
>
>> Looks like it's arena allocation. Basically it's nothing to worry about
>> since virtual address space on 64bit machines is not in short supply. You
>> could limit it with a parameter like so:
>>
>>
>> https://github.com/apache/hadoop-common/blob/28e6d25d448d53cf17041f1c00d0df056f2a1dc0/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh#L128
>>
>>
>> On Tue, Jan 21, 2014 at 3:38 AM, Shangan Chen <[EMAIL PROTECTED]>wrote:
>>
>>> I have quite a lot of flume-agents streaming logs to several
>>> flume-collectors. The problem I face is the memory consumed by
>>> flume-collector is increasing all the time but slowly even I limit the max
>>> heap. I know flume use NIO, but I don't know why it cause memory increase
>>> boundlessly. I prints some information I can get, I really appreciate if
>>> anyone can give any help.
>>>
>>> *my jvm conf:*
>>>
>>> JAVA_OPTS="-Xms8192m -Xmx8192m -Dcom.sun.management.jmxremote
>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/sankuai/logs"
>>>
>>> *top prints of flume instance:*
>>>
>>> top - 17:28:59 up 103 days, 34 min,  4 users,  load average: 0.82, 0.86,
>>> 0.79
>>> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
>>> Cpu(s):  5.4%us,  4.7%sy,  0.0%ni, 89.3%id,  0.0%wa,  0.0%hi,  0.1%si,
>>>  0.4%st
>>> Mem:  16435540k total, 16194572k used,   240968k free,    25896k buffers
>>> Swap:  8385892k total,   205592k used,  8180300k free,   165968k cached
>>>
>>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>> 16580 sankuai   20   0 23.1g  14g 5464 S   92 93.7   7522:53 java
>>>
>>>
>>> *jmap prints :*
>>>
>>> jmap -heap 16580
>>> Attaching to process ID 16580, please wait...
>>> Debugger attached successfully.
>>> Server compiler detected.
>>> JVM version is 23.21-b01
>>>
>>> using thread-local object allocation.
>>> Parallel GC with 8 thread(s)
>>>
>>> Heap Configuration:
>>>    MinHeapFreeRatio = 40
>>>    MaxHeapFreeRatio = 70
>>>    MaxHeapSize      = 8589934592 (8192.0MB)
>>>    NewSize          = 1310720 (1.25MB)
>>>    MaxNewSize       = 17592186044415 MB
>>>    OldSize          = 5439488 (5.1875MB)
>>>    NewRatio         = 2
>>>    SurvivorRatio    = 8
>>>    PermSize         = 21757952 (20.75MB)
>>>    MaxPermSize      = 85983232 (82.0MB)
>>>    G1HeapRegionSize = 0 (0.0MB)
>>>
>>> Heap Usage:
>>> PS Young Generation
>>> Eden Space:
>>>    capacity = 2105147392 (2007.625MB)
>>>    used     = 1048958560 (1000.3648376464844MB)
>>>    free     = 1056188832 (1007.2601623535156MB)
>>>    49.828271596861185% used
>>> From Space:
>>>    capacity = 369229824 (352.125MB)
>>>    used     = 369168400 (352.06642150878906MB)
>>>    free     = 61424 (0.0585784912109375MB)
>>>    99.98336429074591% used
>>> To Space:
>>>    capacity = 388890624 (370.875MB)
>>>    used     = 0 (0.0MB)
>>>    free     = 388890624 (370.875MB)
>>>    0.0% used
>>> PS Old Generation
>>>    capacity = 5726666752 (5461.375MB)
>>>    used     = 4282458544 (4084.0707244873047MB)
>>>    free     = 1444208208 (1377.3042755126953MB)
>>>    74.78099790780352% used
>>> PS Perm Generation
>>>    capacity = 37879808 (36.125MB)
>>>    used     = 37833736 (36.08106231689453MB)
>>>    free     = 46072 (0.04393768310546875MB)
>>>    99.8783731955558% used
>>>
>>> 11441 interned Strings occupying 1025920 bytes.
>>>
>>> *DirectMemory prints(refer https://gist.github.com/rednaxelafx/1593521
>>> <https://gist.github.com/rednaxelafx/1593521>) :*
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB