Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> ZooKeeper Memory Usage


Copy link to this message
-
Re: ZooKeeper Memory Usage
Was there ever a JIRA created for this issue ?

Thanks,
Neha

On Fri, Feb 10, 2012 at 9:31 AM, Mahadev Konar <[EMAIL PROTECTED]> wrote:
> Great. You should have the gc logs then. Mind creating a jira and
> uploading to it?
>
> mahadev
>
> 2012/2/10 César Álvarez Núñez <[EMAIL PROTECTED]>:
>> This is the java.env file content.
>>
>> now=`date +%d%m%Y_%H%M%S`
>>
>> gcLogFile="/srv/zk/GC/`hostname`-${now}.log"
>> gcOpts="${gcOpts} -Xloggc:$gcLogFile"
>> gcOpts="${gcOpts} -XX:+PrintGC"
>> gcOpts="${gcOpts} -XX:+PrintGCTimeStamps"
>> gcOpts="${gcOpts} -XX:+PrintGCDetails"
>> gcOpts="${gcOpts} -XX:+PrintTenuringDistribution"
>> gcOpts="${gcOpts} -XX:+PrintHeapAtGC"
>>
>> gcOpts="${gcOpts} -XX:+AggressiveHeap"
>> #https://cwiki.apache.org/confluence/display/ZOOKEEPER/Troubleshooting
>> #gcOpts="${gcOpts} -XX:+UseConcMarkSweepGC"
>> #gcOpts="${gcOpts} -XX:ParallelGCThreads=8"
>> #gcOpts="-Xms128M -Xmx1G"
>>
>> jvmOpts="${jvmOpts} -d64"
>> jvmOpts="${jvmOpts} -server"
>> jvmOpts="${jvmOpts} -XX:+UseCompressedOops"
>>
>> JVMFLAGS="${gcOpts} ${jvmOpts}"
>>
>> /César.
>>
>> On Thu, Feb 9, 2012 at 7:17 PM, Mahadev Konar <[EMAIL PROTECTED]>wrote:
>>
>>> This is interesting and important.
>>>
>>> Cesar, what jvm options are you running with? Can you the options in:
>>>
>>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Troubleshooting
>>>
>>> Atleast get the GC logs that we can look at?
>>>
>>> This will be very interesting.
>>>
>>> mahadev
>>>
>>>
>>> 2012/2/9 César Álvarez Núñez <[EMAIL PROTECTED]>:
>>> > In my case, our stress test show up a linear increase of "tenured memory"
>>> > from 0 to > 3GiB with ZK 3.4.0 whereas the same stress-test with 3.3.3
>>> > keeps "tenured memory" stable and < 10MiB.
>>> >
>>> > The stress test performs many zNodes creation and delete but the overall
>>> zk
>>> > usage at any moment in time was relative small.
>>> >
>>> > BR,
>>> > /César.
>>> >
>>> > On Thu, Feb 9, 2012 at 3:14 PM, Camille Fournier <[EMAIL PROTECTED]>
>>> wrote:
>>> >
>>> >> This is really a question about how the jvm grows its heaps and resizes
>>> >> them. If the jvm cannot allocate enough memory for the process because
>>> you
>>> >> didn't set the max memory high enough, it will fall over. Zookeeper
>>> keeps
>>> >> its entire state in memory for performance reasons, if it were to swap
>>> that
>>> >> would be quite bad for performance.
>>> >>
>>> >> C
>>> >> On Feb 8, 2012 8:23 PM, "Mike Schilli" <[EMAIL PROTECTED]> wrote:
>>> >>
>>> >> > We've got a ZooKeeper instance that's using about 5 GB of resident
>>> >> > memory. Every time we restart it, it starts at 200MB, and then grows
>>> >> > slowly until it is back at 5 GB.
>>> >> >
>>> >> > The large footprint is related to how much data we've got in there.
>>> >> > What's interesting, though, is that the process size doesn't shrink if
>>> >> > we purge some of the data.
>>> >> >
>>> >> > Now, this isn't a big problem, I'm just curious if the process will
>>> fall
>>> >> > over at some point if it can't get more memory or if it'll just make
>>> due
>>> >> > by caching less data.
>>> >> >
>>> >> > Also, if I remember correctly, there's a configuration variable to set
>>> >> > the maximum size, what happens if ZK reaches that?
>>> >> >
>>> >> > -- -- Mike
>>> >> >
>>> >> > Mike Schilli
>>> >> > [EMAIL PROTECTED]
>>> >> >
>>> >>
>>>
>>>
>>>
>>> --
>>> Mahadev Konar
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>
>
>
> --
> Mahadev Konar
> Hortonworks Inc.
> http://hortonworks.com/
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB