Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> ZooKeeper Memory Usage


Copy link to this message
-
Re: ZooKeeper Memory Usage
Great. You should have the gc logs then. Mind creating a jira and
uploading to it?

mahadev

2012/2/10 César Álvarez Núñez <[EMAIL PROTECTED]>:
> This is the java.env file content.
>
> now=`date +%d%m%Y_%H%M%S`
>
> gcLogFile="/srv/zk/GC/`hostname`-${now}.log"
> gcOpts="${gcOpts} -Xloggc:$gcLogFile"
> gcOpts="${gcOpts} -XX:+PrintGC"
> gcOpts="${gcOpts} -XX:+PrintGCTimeStamps"
> gcOpts="${gcOpts} -XX:+PrintGCDetails"
> gcOpts="${gcOpts} -XX:+PrintTenuringDistribution"
> gcOpts="${gcOpts} -XX:+PrintHeapAtGC"
>
> gcOpts="${gcOpts} -XX:+AggressiveHeap"
> #https://cwiki.apache.org/confluence/display/ZOOKEEPER/Troubleshooting
> #gcOpts="${gcOpts} -XX:+UseConcMarkSweepGC"
> #gcOpts="${gcOpts} -XX:ParallelGCThreads=8"
> #gcOpts="-Xms128M -Xmx1G"
>
> jvmOpts="${jvmOpts} -d64"
> jvmOpts="${jvmOpts} -server"
> jvmOpts="${jvmOpts} -XX:+UseCompressedOops"
>
> JVMFLAGS="${gcOpts} ${jvmOpts}"
>
> /César.
>
> On Thu, Feb 9, 2012 at 7:17 PM, Mahadev Konar <[EMAIL PROTECTED]>wrote:
>
>> This is interesting and important.
>>
>> Cesar, what jvm options are you running with? Can you the options in:
>>
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Troubleshooting
>>
>> Atleast get the GC logs that we can look at?
>>
>> This will be very interesting.
>>
>> mahadev
>>
>>
>> 2012/2/9 César Álvarez Núñez <[EMAIL PROTECTED]>:
>> > In my case, our stress test show up a linear increase of "tenured memory"
>> > from 0 to > 3GiB with ZK 3.4.0 whereas the same stress-test with 3.3.3
>> > keeps "tenured memory" stable and < 10MiB.
>> >
>> > The stress test performs many zNodes creation and delete but the overall
>> zk
>> > usage at any moment in time was relative small.
>> >
>> > BR,
>> > /César.
>> >
>> > On Thu, Feb 9, 2012 at 3:14 PM, Camille Fournier <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> This is really a question about how the jvm grows its heaps and resizes
>> >> them. If the jvm cannot allocate enough memory for the process because
>> you
>> >> didn't set the max memory high enough, it will fall over. Zookeeper
>> keeps
>> >> its entire state in memory for performance reasons, if it were to swap
>> that
>> >> would be quite bad for performance.
>> >>
>> >> C
>> >> On Feb 8, 2012 8:23 PM, "Mike Schilli" <[EMAIL PROTECTED]> wrote:
>> >>
>> >> > We've got a ZooKeeper instance that's using about 5 GB of resident
>> >> > memory. Every time we restart it, it starts at 200MB, and then grows
>> >> > slowly until it is back at 5 GB.
>> >> >
>> >> > The large footprint is related to how much data we've got in there.
>> >> > What's interesting, though, is that the process size doesn't shrink if
>> >> > we purge some of the data.
>> >> >
>> >> > Now, this isn't a big problem, I'm just curious if the process will
>> fall
>> >> > over at some point if it can't get more memory or if it'll just make
>> due
>> >> > by caching less data.
>> >> >
>> >> > Also, if I remember correctly, there's a configuration variable to set
>> >> > the maximum size, what happens if ZK reaches that?
>> >> >
>> >> > -- -- Mike
>> >> >
>> >> > Mike Schilli
>> >> > [EMAIL PROTECTED]
>> >> >
>> >>
>>
>>
>>
>> --
>> Mahadev Konar
>> Hortonworks Inc.
>> http://hortonworks.com/
>>

--
Mahadev Konar
Hortonworks Inc.
http://hortonworks.com/