Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> flume use quite a lot of direct momery and blast


Copy link to this message
-
Re: flume use quite a lot of direct momery and blast
Looks like it's arena allocation. Basically it's nothing to worry about
since virtual address space on 64bit machines is not in short supply. You
could limit it with a parameter like so:

https://github.com/apache/hadoop-common/blob/28e6d25d448d53cf17041f1c00d0df056f2a1dc0/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh#L128
On Tue, Jan 21, 2014 at 3:38 AM, Shangan Chen <[EMAIL PROTECTED]>wrote:

> I have quite a lot of flume-agents streaming logs to several
> flume-collectors. The problem I face is the memory consumed by
> flume-collector is increasing all the time but slowly even I limit the max
> heap. I know flume use NIO, but I don't know why it cause memory increase
> boundlessly. I prints some information I can get, I really appreciate if
> anyone can give any help.
>
> *my jvm conf:*
>
> JAVA_OPTS="-Xms8192m -Xmx8192m -Dcom.sun.management.jmxremote
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/sankuai/logs"
>
> *top prints of flume instance:*
>
> top - 17:28:59 up 103 days, 34 min,  4 users,  load average: 0.82, 0.86,
> 0.79
> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
> Cpu(s):  5.4%us,  4.7%sy,  0.0%ni, 89.3%id,  0.0%wa,  0.0%hi,  0.1%si,
>  0.4%st
> Mem:  16435540k total, 16194572k used,   240968k free,    25896k buffers
> Swap:  8385892k total,   205592k used,  8180300k free,   165968k cached
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 16580 sankuai   20   0 23.1g  14g 5464 S   92 93.7   7522:53 java
>
>
> *jmap prints :*
>
> jmap -heap 16580
> Attaching to process ID 16580, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 23.21-b01
>
> using thread-local object allocation.
> Parallel GC with 8 thread(s)
>
> Heap Configuration:
>    MinHeapFreeRatio = 40
>    MaxHeapFreeRatio = 70
>    MaxHeapSize      = 8589934592 (8192.0MB)
>    NewSize          = 1310720 (1.25MB)
>    MaxNewSize       = 17592186044415 MB
>    OldSize          = 5439488 (5.1875MB)
>    NewRatio         = 2
>    SurvivorRatio    = 8
>    PermSize         = 21757952 (20.75MB)
>    MaxPermSize      = 85983232 (82.0MB)
>    G1HeapRegionSize = 0 (0.0MB)
>
> Heap Usage:
> PS Young Generation
> Eden Space:
>    capacity = 2105147392 (2007.625MB)
>    used     = 1048958560 (1000.3648376464844MB)
>    free     = 1056188832 (1007.2601623535156MB)
>    49.828271596861185% used
> From Space:
>    capacity = 369229824 (352.125MB)
>    used     = 369168400 (352.06642150878906MB)
>    free     = 61424 (0.0585784912109375MB)
>    99.98336429074591% used
> To Space:
>    capacity = 388890624 (370.875MB)
>    used     = 0 (0.0MB)
>    free     = 388890624 (370.875MB)
>    0.0% used
> PS Old Generation
>    capacity = 5726666752 (5461.375MB)
>    used     = 4282458544 (4084.0707244873047MB)
>    free     = 1444208208 (1377.3042755126953MB)
>    74.78099790780352% used
> PS Perm Generation
>    capacity = 37879808 (36.125MB)
>    used     = 37833736 (36.08106231689453MB)
>    free     = 46072 (0.04393768310546875MB)
>    99.8783731955558% used
>
> 11441 interned Strings occupying 1025920 bytes.
>
> *DirectMemory prints(refer https://gist.github.com/rednaxelafx/1593521
> <https://gist.github.com/rednaxelafx/1593521>) :*
>
> java -cp .:$JAVA_HOME/lib/sa-jdi.jar DirectMemorySize 16580
> Attaching to process ID 16580, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 23.21-b01
> NIO direct memory: (in bytes)
>   reserved size = 5864.191787 MB (6149050767 bytes)
>   max size      = 7850.687500 MB (8232042496 bytes)
>
>
>
> --
> have a good day!
> chenshang'an
>
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB