Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> setting zookeeper heap size


Copy link to this message
-
Re: setting zookeeper heap size
The problem in my mind is that it raises the spectre of users
overcommitting their servers accidentally, and instead of seeing the
result as a hard fail if their VM runs out of available memory, they
will see it as a useless system when everything starts swapping to
disk because the total heap size for VMs on the machine has grown
beyond the reasonably allocatable physical memory of the machine.
I don't really care about the ergonomics of client VMs, but server VMs
in 64bit servers defaulting to 1/4 the memory of the system seems
excessive. Anyway, whether it is reasonable for the vast case of VMs
out there in the wild or not, it is way overboard for the vast case of
ZooKeepers out there in the wild. I think we should consider offering
guidance on this in documentation or the scripts.

C

On Thu, Nov 10, 2011 at 11:59 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:
> The behavior without xmx is actually pretty reasonable.  Note that the
> default of half is only up to a few hundred MB of memory (not GB).  Even
> when lots of memory is allocated, the jvm avoids expanding the actual use
> beyond reasonable need.
>
> On Thu, Nov 10, 2011 at 8:35 PM, Camille Fournier <[EMAIL PROTECTED]>wrote:
>
>> Btw I'm pretty sure res is used heap and virt is just mapped but yet to be
>> filled. Use jstat -gc to see actual usage for the various collectors.
>>
>> Also, if it's true that the jvm defaults to a xmx of half the ram (wtf bad
>> decision) we had better ship zk start scripts with more reasonable xmx
>> configured.
>> Although I've used u18 and u23 extensively on 64bit servers and never seen
>> it default to half the physical memory, or 32gig.
>>
>> C
>>
>> From my phone
>> On Nov 10, 2011 6:50 PM, "Ed Sexton" <[EMAIL PROTECTED]> wrote:
>>
>> > Hi Henry,
>> >
>> > I checked the process listing via ps and top, and saw VIRT=6477m  and
>> > RES=83m.
>> >
>> > How would one check if it is the jvm heap vs mapped space?
>> >
>> > Thanks as always Henry.
>> > Ed
>> >
>> > On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <[EMAIL PROTECTED]>
>> > wrote:
>> >
>> > > Ed - how are you making the observation that the heap size is 6477m? Is
>> > it
>> > > from a profiler, or from top or similar? The amount of virtual memory
>> > > mapped for Java processes can often be very large, especially on recent
>> > > RHEL systems with the new arena allocator in glibc. Worth checking
>> > whether
>> > > it's the JVM heap or the amount of mapped space.
>> > >
>> > > cheers,
>> > > Henry
>> > >
>> > > On 10 November 2011 13:47, Ed Sexton <[EMAIL PROTECTED]> wrote:
>> > >
>> > > > Hello Group-
>> > > >
>> > > > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
>> > zookeeper
>> > > > is running with a heapsize of 6477m.
>> > > >
>> > > > I start zookeeper standalone, via
>> /etc/init.d/hadoop-zookeeper-server.
>> >  I
>> > > > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
>> > zoo.cfg,
>> > > > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
>> > > which
>> > > > is what is called by the startup script, it just has this definition:
>> > > >
>> > > > JVMFLAGS=-Dzookeeper.log.threshold=INFO
>> > > >
>> > > > Can someone guide me to where the 6GB heapsize is being referenced?
>> >  This
>> > > > seems a bit large of a heap.
>> > > >
>> > > > Thanks for your guidance.
>> > > >
>> > > > Sincerely,
>> > > > Ed
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Henry Robinson
>> > > Software Engineer
>> > > Cloudera
>> > > 415-994-6679
>> > >
>> >
>> >
>> >
>> > --
>> > Sincerely,
>> > Ed Sexton
>> > e: [EMAIL PROTECTED]
>> > gv: (408) 475-2358
>> >
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB