Hello Group-

I am running CDH zookeeper version 3.3.3+12.12 and noticed that zookeeper
is running with a heapsize of 6477m.

I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server.  I
do not see reference to any ZOOKEEPER_HEAPIZE in the startup or zoo.cfg,
nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server, which
is what is called by the startup script, it just has this definition:

JVMFLAGS=-Dzookeeper.log.threshold=INFO

Can someone guide me to where the 6GB heapsize is being referenced?  This
seems a bit large of a heap.

Thanks for your guidance.

Sincerely,
Ed

In general ZK does not bound the heap size. The default is to use
whatever the JVM prefers, which is typically based on available host
memory. You can override this using JVMFLAGS.

Patrick

On Thu, Nov 10, 2011 at 1:47 PM, Ed Sexton <[EMAIL PROTECTED]> wrote:
> Hello Group-
>
> I am running CDH zookeeper version 3.3.3+12.12 and noticed that zookeeper
> is running with a heapsize of 6477m.
>
> I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server.  I
> do not see reference to any ZOOKEEPER_HEAPIZE in the startup or zoo.cfg,
> nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server, which
> is what is called by the startup script, it just has this definition:
>
> JVMFLAGS=-Dzookeeper.log.threshold=INFO
>
> Can someone guide me to where the 6GB heapsize is being referenced?  This
> seems a bit large of a heap.
>
> Thanks for your guidance.
>
> Sincerely,
> Ed
>

Ed - how are you making the observation that the heap size is 6477m? Is it
from a profiler, or from top or similar? The amount of virtual memory
mapped for Java processes can often be very large, especially on recent
RHEL systems with the new arena allocator in glibc. Worth checking whether
it's the JVM heap or the amount of mapped space.

cheers,
Henry

On 10 November 2011 13:47, Ed Sexton <[EMAIL PROTECTED]> wrote:

> Hello Group-
>
> I am running CDH zookeeper version 3.3.3+12.12 and noticed that zookeeper
> is running with a heapsize of 6477m.
>
> I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server.  I
> do not see reference to any ZOOKEEPER_HEAPIZE in the startup or zoo.cfg,
> nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server, which
> is what is called by the startup script, it just has this definition:
>
> JVMFLAGS=-Dzookeeper.log.threshold=INFO
>
> Can someone guide me to where the 6GB heapsize is being referenced?  This
> seems a bit large of a heap.
>
> Thanks for your guidance.
>
> Sincerely,
> Ed
>

--
Henry Robinson
Software Engineer
Cloudera
415-994-6679

Hi Henry,

I checked the process listing via ps and top, and saw VIRT=6477m  and
RES=83m.

How would one check if it is the jvm heap vs mapped space?

Thanks as always Henry.
Ed

On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <[EMAIL PROTECTED]> wrote:

> Ed - how are you making the observation that the heap size is 6477m? Is it
> from a profiler, or from top or similar? The amount of virtual memory
> mapped for Java processes can often be very large, especially on recent
> RHEL systems with the new arena allocator in glibc. Worth checking whether
> it's the JVM heap or the amount of mapped space.
>
> cheers,
> Henry
>
> On 10 November 2011 13:47, Ed Sexton <[EMAIL PROTECTED]> wrote:
>
> > Hello Group-
> >
> > I am running CDH zookeeper version 3.3.3+12.12 and noticed that zookeeper
> > is running with a heapsize of 6477m.
> >
> > I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server.  I
> > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or zoo.cfg,
> > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
> which
> > is what is called by the startup script, it just has this definition:
> >
> > JVMFLAGS=-Dzookeeper.log.threshold=INFO
> >
> > Can someone guide me to where the 6GB heapsize is being referenced?  This
> > seems a bit large of a heap.
> >
> > Thanks for your guidance.
> >
> > Sincerely,
> > Ed
> >
>
>
>
> --
> Henry Robinson
> Software Engineer
> Cloudera
> 415-994-6679
>

--
Sincerely,
Ed Sexton
e: [EMAIL PROTECTED]
gv: (408) 475-2358

It's very odd for the jvm to chose a 6g heap. On a 64bit machine it might
choose 2g, I've never seen it go larger without xmx. Check to see if you
have some sort of flags unintentionally added.

C
>From my phone
On Nov 10, 2011 4:59 PM, "Patrick Hunt" <[EMAIL PROTECTED]> wrote:

> In general ZK does not bound the heap size. The default is to use
> whatever the JVM prefers, which is typically based on available host
> memory. You can override this using JVMFLAGS.
>
> Patrick
>
> On Thu, Nov 10, 2011 at 1:47 PM, Ed Sexton <[EMAIL PROTECTED]> wrote:
> > Hello Group-
> >
> > I am running CDH zookeeper version 3.3.3+12.12 and noticed that zookeeper
> > is running with a heapsize of 6477m.
> >
> > I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server.  I
> > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or zoo.cfg,
> > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
> which
> > is what is called by the startup script, it just has this definition:
> >
> > JVMFLAGS=-Dzookeeper.log.threshold=INFO
> >
> > Can someone guide me to where the 6GB heapsize is being referenced?  This
> > seems a bit large of a heap.
> >
> > Thanks for your guidance.
> >
> > Sincerely,
> > Ed
> >
>

It changed "recently":
http://www.oracle.com/technetwork/java/javase/6u18-142093.html

"The default maximum heap size is half of the physical memory up to a
physical memory size of 192 megabytes and otherwise one fourth of the
physical memory up to a physical memory size of 1 gigabyte. "
"Server JVM heap configuration ergonomics are now the same as the
Client, except that the default maximum heap size for 32-bit JVMs is 1
gigabyte, corresponding to a physical memory size of 4 gigabytes, and
for 64-bit JVMs is 32 gigabytes, corresponding to a physical memory
size of 128 gigabytes."

Patrick

On Thu, Nov 10, 2011 at 4:14 PM, Camille Fournier <[EMAIL PROTECTED]> wrote:
> It's very odd for the jvm to chose a 6g heap. On a 64bit machine it might
> choose 2g, I've never seen it go larger without xmx. Check to see if you
> have some sort of flags unintentionally added.
>
> C
> From my phone
> On Nov 10, 2011 4:59 PM, "Patrick Hunt" <[EMAIL PROTECTED]> wrote:
>
>> In general ZK does not bound the heap size. The default is to use
>> whatever the JVM prefers, which is typically based on available host
>> memory. You can override this using JVMFLAGS.
>>
>> Patrick
>>
>> On Thu, Nov 10, 2011 at 1:47 PM, Ed Sexton <[EMAIL PROTECTED]> wrote:
>> > Hello Group-
>> >
>> > I am running CDH zookeeper version 3.3.3+12.12 and noticed that zookeeper
>> > is running with a heapsize of 6477m.
>> >
>> > I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server.  I
>> > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or zoo.cfg,
>> > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
>> which
>> > is what is called by the startup script, it just has this definition:
>> >
>> > JVMFLAGS=-Dzookeeper.log.threshold=INFO
>> >
>> > Can someone guide me to where the 6GB heapsize is being referenced?  This
>> > seems a bit large of a heap.
>> >
>> > Thanks for your guidance.
>> >
>> > Sincerely,
>> > Ed
>> >
>>
>

Perhaps easier to just use jmap?

sudo jmap -heap <jvmpid>

Patrick

On Thu, Nov 10, 2011 at 3:49 PM, Ed Sexton <[EMAIL PROTECTED]> wrote:
> Hi Henry,
>
> I checked the process listing via ps and top, and saw VIRT=6477m  and
> RES=83m.
>
> How would one check if it is the jvm heap vs mapped space?
>
> Thanks as always Henry.
> Ed
>
> On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <[EMAIL PROTECTED]> wrote:
>
>> Ed - how are you making the observation that the heap size is 6477m? Is it
>> from a profiler, or from top or similar? The amount of virtual memory
>> mapped for Java processes can often be very large, especially on recent
>> RHEL systems with the new arena allocator in glibc. Worth checking whether
>> it's the JVM heap or the amount of mapped space.
>>
>> cheers,
>> Henry
>>
>> On 10 November 2011 13:47, Ed Sexton <[EMAIL PROTECTED]> wrote:
>>
>> > Hello Group-
>> >
>> > I am running CDH zookeeper version 3.3.3+12.12 and noticed that zookeeper
>> > is running with a heapsize of 6477m.
>> >
>> > I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server.  I
>> > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or zoo.cfg,
>> > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
>> which
>> > is what is called by the startup script, it just has this definition:
>> >
>> > JVMFLAGS=-Dzookeeper.log.threshold=INFO
>> >
>> > Can someone guide me to where the 6GB heapsize is being referenced?  This
>> > seems a bit large of a heap.
>> >
>> > Thanks for your guidance.
>> >
>> > Sincerely,
>> > Ed
>> >
>>
>>
>>
>> --
>> Henry Robinson
>> Software Engineer
>> Cloudera
>> 415-994-6679
>>
>
>
>
> --
> Sincerely,
> Ed Sexton
> e: [EMAIL PROTECTED]
> gv: (408) 475-2358
>

If you are using RHEL 6 chances are you're running into the newer per
thread allocator in recent glibc. See
http://www.quora.com/Why-do-some-applications-use-significantly-more-virtual-memory-on-RHEL-6-compared-to-RHEL-5

If so, it's harmless. Virtual memory address space is not a scarce
resource. If it concerns you, you can set MALLOC_ARENA_MAX to something
like 4 in the JVM's environment.

Henry

On Thursday, 10 November 2011, Camille Fournier <[EMAIL PROTECTED]> wrote:
> It's very odd for the jvm to chose a 6g heap. On a 64bit machine it might
> choose 2g, I've never seen it go larger without xmx. Check to see if you
> have some sort of flags unintentionally added.
>
> C
> From my phone
> On Nov 10, 2011 4:59 PM, "Patrick Hunt" <[EMAIL PROTECTED]> wrote:
>
>> In general ZK does not bound the heap size. The default is to use
>> whatever the JVM prefers, which is typically based on available host
>> memory. You can override this using JVMFLAGS.
>>
>> Patrick
>>
>> On Thu, Nov 10, 2011 at 1:47 PM, Ed Sexton <[EMAIL PROTECTED]>
wrote:
>> > Hello Group-
>> >
>> > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
zookeeper
>> > is running with a heapsize of 6477m.
>> >
>> > I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server.
 I
>> > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
zoo.cfg,
>> > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
>> which
>> > is what is called by the startup script, it just has this definition:
>> >
>> > JVMFLAGS=-Dzookeeper.log.threshold=INFO
>> >
>> > Can someone guide me to where the 6GB heapsize is being referenced?
 This
>> > seems a bit large of a heap.
>> >
>> > Thanks for your guidance.
>> >
>> > Sincerely,
>> > Ed
>> >
>>
>

--
Henry Robinson
Software Engineer
Cloudera
415-994-6679

Thanks all, I am understanding things a bit better.

Here is the output of the heap.  1/4 of the available memory (24gb) does
put it at ~6gb.

I will override using the JVMFLAG and test it out.  If this is not a wise
thing to do, please let me know - thanks.

- Ed

# /usr/java/jdk1.6.0_25/bin/jmap -heap 11508
Attaching to process ID 11508, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.0-b11

using thread-local object allocation.
Parallel GC with 13 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 6318718976 (6026.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 85983232 (82.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 262144 (0.25MB)
   used     = 132088 (0.12596893310546875MB)
   free     = 130056 (0.12403106689453125MB)
   50.3875732421875% used
>From Space:
   capacity = 1900544 (1.8125MB)
   used     = 163840 (0.15625MB)
   free     = 1736704 (1.65625MB)
   8.620689655172415% used
To Space:
   capacity = 1900544 (1.8125MB)
   used     = 0 (0.0MB)
   free     = 1900544 (1.8125MB)
   0.0% used
PS Old Generation
   capacity = 4587520 (4.375MB)
   used     = 2752448 (2.62493896484375MB)
   free     = 1835072 (1.75006103515625MB)
   59.998604910714285% used
PS Perm Generation
   capacity = 21757952 (20.75MB)
   used     = 10180360 (9.708747863769531MB)
   free     = 11577592 (11.041252136230469MB)
   46.78914633141943% used
On Thu, Nov 10, 2011 at 4:51 PM, Patrick Hunt <[EMAIL PROTECTED]> wrote:

> Perhaps easier to just use jmap?
>
> sudo jmap -heap <jvmpid>
>
> Patrick
>
> On Thu, Nov 10, 2011 at 3:49 PM, Ed Sexton <[EMAIL PROTECTED]> wrote:
> > Hi Henry,
> >
> > I checked the process listing via ps and top, and saw VIRT=6477m  and
> > RES=83m.
> >
> > How would one check if it is the jvm heap vs mapped space?
> >
> > Thanks as always Henry.
> > Ed
> >
> > On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <[EMAIL PROTECTED]>
> wrote:
> >
> >> Ed - how are you making the observation that the heap size is 6477m? Is
> it
> >> from a profiler, or from top or similar? The amount of virtual memory
> >> mapped for Java processes can often be very large, especially on recent
> >> RHEL systems with the new arena allocator in glibc. Worth checking
> whether
> >> it's the JVM heap or the amount of mapped space.
> >>
> >> cheers,
> >> Henry
> >>
> >> On 10 November 2011 13:47, Ed Sexton <[EMAIL PROTECTED]> wrote:
> >>
> >> > Hello Group-
> >> >
> >> > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
> zookeeper
> >> > is running with a heapsize of 6477m.
> >> >
> >> > I start zookeeper standalone, via
> /etc/init.d/hadoop-zookeeper-server.  I
> >> > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
> zoo.cfg,
> >> > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
> >> which
> >> > is what is called by the startup script, it just has this definition:
> >> >
> >> > JVMFLAGS=-Dzookeeper.log.threshold=INFO
> >> >
> >> > Can someone guide me to where the 6GB heapsize is being referenced?
>  This
> >> > seems a bit large of a heap.
> >> >
> >> > Thanks for your guidance.
> >> >
> >> > Sincerely,
> >> > Ed
> >> >
> >>
> >>
> >>
> >> --
> >> Henry Robinson
> >> Software Engineer
> >> Cloudera
> >> 415-994-6679
> >>
> >
> >
> >
>

Btw I'm pretty sure res is used heap and virt is just mapped but yet to be
filled. Use jstat -gc to see actual usage for the various collectors.

Also, if it's true that the jvm defaults to a xmx of half the ram (wtf bad
decision) we had better ship zk start scripts with more reasonable xmx
configured.
Although I've used u18 and u23 extensively on 64bit servers and never seen
it default to half the physical memory, or 32gig.

C

>From my phone
On Nov 10, 2011 6:50 PM, "Ed Sexton" <[EMAIL PROTECTED]> wrote:

> Hi Henry,
>
> I checked the process listing via ps and top, and saw VIRT=6477m  and
> RES=83m.
>
> How would one check if it is the jvm heap vs mapped space?
>
> Thanks as always Henry.
> Ed
>
> On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <[EMAIL PROTECTED]>
> wrote:
>
> > Ed - how are you making the observation that the heap size is 6477m? Is
> it
> > from a profiler, or from top or similar? The amount of virtual memory
> > mapped for Java processes can often be very large, especially on recent
> > RHEL systems with the new arena allocator in glibc. Worth checking
> whether
> > it's the JVM heap or the amount of mapped space.
> >
> > cheers,
> > Henry
> >
> > On 10 November 2011 13:47, Ed Sexton <[EMAIL PROTECTED]> wrote:
> >
> > > Hello Group-
> > >
> > > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
> zookeeper
> > > is running with a heapsize of 6477m.
> > >
> > > I start zookeeper standalone, via /etc/init.d/hadoop-zookeeper-server.
>  I
> > > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
> zoo.cfg,
> > > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
> > which
> > > is what is called by the startup script, it just has this definition:
> > >
> > > JVMFLAGS=-Dzookeeper.log.threshold=INFO
> > >
> > > Can someone guide me to where the 6GB heapsize is being referenced?
>  This
> > > seems a bit large of a heap.
> > >
> > > Thanks for your guidance.
> > >
> > > Sincerely,
> > > Ed
> > >
> >
> >
> >
> > --
> > Henry Robinson
> > Software Engineer
> > Cloudera
> > 415-994-6679
> >
>
>
>
> --
> Sincerely,
> Ed Sexton
> e: [EMAIL PROTECTED]
> gv: (408) 475-2358
>

Re wiseness - that all depends on the size of the content you are storing. ;-)

You could get more insight using jmap and the like to get an idea of
how much mem you are using. (in a test setup say)

Patrick

On Thu, Nov 10, 2011 at 5:32 PM, Ed Sexton <[EMAIL PROTECTED]> wrote:
> Thanks all, I am understanding things a bit better.
>
> Here is the output of the heap.  1/4 of the available memory (24gb) does
> put it at ~6gb.
>
> I will override using the JVMFLAG and test it out.  If this is not a wise
> thing to do, please let me know - thanks.
>
> - Ed
>
> # /usr/java/jdk1.6.0_25/bin/jmap -heap 11508
> Attaching to process ID 11508, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 20.0-b11
>
> using thread-local object allocation.
> Parallel GC with 13 thread(s)
>
> Heap Configuration:
>   MinHeapFreeRatio = 40
>   MaxHeapFreeRatio = 70
>   MaxHeapSize      = 6318718976 (6026.0MB)
>   NewSize          = 1310720 (1.25MB)
>   MaxNewSize       = 17592186044415 MB
>   OldSize          = 5439488 (5.1875MB)
>   NewRatio         = 2
>   SurvivorRatio    = 8
>   PermSize         = 21757952 (20.75MB)
>   MaxPermSize      = 85983232 (82.0MB)
>
> Heap Usage:
> PS Young Generation
> Eden Space:
>   capacity = 262144 (0.25MB)
>   used     = 132088 (0.12596893310546875MB)
>   free     = 130056 (0.12403106689453125MB)
>   50.3875732421875% used
> From Space:
>   capacity = 1900544 (1.8125MB)
>   used     = 163840 (0.15625MB)
>   free     = 1736704 (1.65625MB)
>   8.620689655172415% used
> To Space:
>   capacity = 1900544 (1.8125MB)
>   used     = 0 (0.0MB)
>   free     = 1900544 (1.8125MB)
>   0.0% used
> PS Old Generation
>   capacity = 4587520 (4.375MB)
>   used     = 2752448 (2.62493896484375MB)
>   free     = 1835072 (1.75006103515625MB)
>   59.998604910714285% used
> PS Perm Generation
>   capacity = 21757952 (20.75MB)
>   used     = 10180360 (9.708747863769531MB)
>   free     = 11577592 (11.041252136230469MB)
>   46.78914633141943% used
>
>
>
>
> On Thu, Nov 10, 2011 at 4:51 PM, Patrick Hunt <[EMAIL PROTECTED]> wrote:
>
>> Perhaps easier to just use jmap?
>>
>> sudo jmap -heap <jvmpid>
>>
>> Patrick
>>
>> On Thu, Nov 10, 2011 at 3:49 PM, Ed Sexton <[EMAIL PROTECTED]> wrote:
>> > Hi Henry,
>> >
>> > I checked the process listing via ps and top, and saw VIRT=6477m  and
>> > RES=83m.
>> >
>> > How would one check if it is the jvm heap vs mapped space?
>> >
>> > Thanks as always Henry.
>> > Ed
>> >
>> > On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> Ed - how are you making the observation that the heap size is 6477m? Is
>> it
>> >> from a profiler, or from top or similar? The amount of virtual memory
>> >> mapped for Java processes can often be very large, especially on recent
>> >> RHEL systems with the new arena allocator in glibc. Worth checking
>> whether
>> >> it's the JVM heap or the amount of mapped space.
>> >>
>> >> cheers,
>> >> Henry
>> >>
>> >> On 10 November 2011 13:47, Ed Sexton <[EMAIL PROTECTED]> wrote:
>> >>
>> >> > Hello Group-
>> >> >
>> >> > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
>> zookeeper
>> >> > is running with a heapsize of 6477m.
>> >> >
>> >> > I start zookeeper standalone, via
>> /etc/init.d/hadoop-zookeeper-server.  I
>> >> > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
>> zoo.cfg,
>> >> > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
>> >> which
>> >> > is what is called by the startup script, it just has this definition:
>> >> >
>> >> > JVMFLAGS=-Dzookeeper.log.threshold=INFO
>> >> >
>> >> > Can someone guide me to where the 6GB heapsize is being referenced?
>>  This
>> >> > seems a bit large of a heap.
>> >> >
>> >> > Thanks for your guidance.
>> >> >
>> >> > Sincerely,
>> >> > Ed
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Henry Robinson
>> >> Software Engineer
>> >> Cloudera
>> >> 415-994-6679
>> >>
>> >
>> >
>> >
>>
>

The behavior without xmx is actually pretty reasonable.  Note that the
default of half is only up to a few hundred MB of memory (not GB).  Even
when lots of memory is allocated, the jvm avoids expanding the actual use
beyond reasonable need.

On Thu, Nov 10, 2011 at 8:35 PM, Camille Fournier <[EMAIL PROTECTED]>wrote:

> Btw I'm pretty sure res is used heap and virt is just mapped but yet to be
> filled. Use jstat -gc to see actual usage for the various collectors.
>
> Also, if it's true that the jvm defaults to a xmx of half the ram (wtf bad
> decision) we had better ship zk start scripts with more reasonable xmx
> configured.
> Although I've used u18 and u23 extensively on 64bit servers and never seen
> it default to half the physical memory, or 32gig.
>
> C
>
> From my phone
> On Nov 10, 2011 6:50 PM, "Ed Sexton" <[EMAIL PROTECTED]> wrote:
>
> > Hi Henry,
> >
> > I checked the process listing via ps and top, and saw VIRT=6477m  and
> > RES=83m.
> >
> > How would one check if it is the jvm heap vs mapped space?
> >
> > Thanks as always Henry.
> > Ed
> >
> > On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Ed - how are you making the observation that the heap size is 6477m? Is
> > it
> > > from a profiler, or from top or similar? The amount of virtual memory
> > > mapped for Java processes can often be very large, especially on recent
> > > RHEL systems with the new arena allocator in glibc. Worth checking
> > whether
> > > it's the JVM heap or the amount of mapped space.
> > >
> > > cheers,
> > > Henry
> > >
> > > On 10 November 2011 13:47, Ed Sexton <[EMAIL PROTECTED]> wrote:
> > >
> > > > Hello Group-
> > > >
> > > > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
> > zookeeper
> > > > is running with a heapsize of 6477m.
> > > >
> > > > I start zookeeper standalone, via
> /etc/init.d/hadoop-zookeeper-server.
> >  I
> > > > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
> > zoo.cfg,
> > > > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
> > > which
> > > > is what is called by the startup script, it just has this definition:
> > > >
> > > > JVMFLAGS=-Dzookeeper.log.threshold=INFO
> > > >
> > > > Can someone guide me to where the 6GB heapsize is being referenced?
> >  This
> > > > seems a bit large of a heap.
> > > >
> > > > Thanks for your guidance.
> > > >
> > > > Sincerely,
> > > > Ed
> > > >
> > >
> > >
> > >
> > > --
> > > Henry Robinson
> > > Software Engineer
> > > Cloudera
> > > 415-994-6679
> > >
> >
> >
> >
> > --
> > Sincerely,
> > Ed Sexton
> > e: [EMAIL PROTECTED]
> > gv: (408) 475-2358
> >
>

The problem in my mind is that it raises the spectre of users
overcommitting their servers accidentally, and instead of seeing the
result as a hard fail if their VM runs out of available memory, they
will see it as a useless system when everything starts swapping to
disk because the total heap size for VMs on the machine has grown
beyond the reasonably allocatable physical memory of the machine.
I don't really care about the ergonomics of client VMs, but server VMs
in 64bit servers defaulting to 1/4 the memory of the system seems
excessive. Anyway, whether it is reasonable for the vast case of VMs
out there in the wild or not, it is way overboard for the vast case of
ZooKeepers out there in the wild. I think we should consider offering
guidance on this in documentation or the scripts.

C

On Thu, Nov 10, 2011 at 11:59 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:
> The behavior without xmx is actually pretty reasonable.  Note that the
> default of half is only up to a few hundred MB of memory (not GB).  Even
> when lots of memory is allocated, the jvm avoids expanding the actual use
> beyond reasonable need.
>
> On Thu, Nov 10, 2011 at 8:35 PM, Camille Fournier <[EMAIL PROTECTED]>wrote:
>
>> Btw I'm pretty sure res is used heap and virt is just mapped but yet to be
>> filled. Use jstat -gc to see actual usage for the various collectors.
>>
>> Also, if it's true that the jvm defaults to a xmx of half the ram (wtf bad
>> decision) we had better ship zk start scripts with more reasonable xmx
>> configured.
>> Although I've used u18 and u23 extensively on 64bit servers and never seen
>> it default to half the physical memory, or 32gig.
>>
>> C
>>
>> From my phone
>> On Nov 10, 2011 6:50 PM, "Ed Sexton" <[EMAIL PROTECTED]> wrote:
>>
>> > Hi Henry,
>> >
>> > I checked the process listing via ps and top, and saw VIRT=6477m  and
>> > RES=83m.
>> >
>> > How would one check if it is the jvm heap vs mapped space?
>> >
>> > Thanks as always Henry.
>> > Ed
>> >
>> > On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <[EMAIL PROTECTED]>
>> > wrote:
>> >
>> > > Ed - how are you making the observation that the heap size is 6477m? Is
>> > it
>> > > from a profiler, or from top or similar? The amount of virtual memory
>> > > mapped for Java processes can often be very large, especially on recent
>> > > RHEL systems with the new arena allocator in glibc. Worth checking
>> > whether
>> > > it's the JVM heap or the amount of mapped space.
>> > >
>> > > cheers,
>> > > Henry
>> > >
>> > > On 10 November 2011 13:47, Ed Sexton <[EMAIL PROTECTED]> wrote:
>> > >
>> > > > Hello Group-
>> > > >
>> > > > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
>> > zookeeper
>> > > > is running with a heapsize of 6477m.
>> > > >
>> > > > I start zookeeper standalone, via
>> /etc/init.d/hadoop-zookeeper-server.
>> >  I
>> > > > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
>> > zoo.cfg,
>> > > > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
>> > > which
>> > > > is what is called by the startup script, it just has this definition:
>> > > >
>> > > > JVMFLAGS=-Dzookeeper.log.threshold=INFO
>> > > >
>> > > > Can someone guide me to where the 6GB heapsize is being referenced?
>> >  This
>> > > > seems a bit large of a heap.
>> > > >
>> > > > Thanks for your guidance.
>> > > >
>> > > > Sincerely,
>> > > > Ed
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Henry Robinson
>> > > Software Engineer
>> > > Cloudera
>> > > 415-994-6679
>> > >
>> >
>> >
>> >
>> > --
>> > Sincerely,
>> > Ed Sexton
>> > e: [EMAIL PROTECTED]
>> > gv: (408) 475-2358
>> >
>>
>