Hi,

I am trying to load a huge matrix(100,000 by 500) to my zookeeper
instance. Each element of the matrix is a ZNODE and value of each
element is a digit(0-9).

But I'm only able to load around 1000 x 500 nodes. Zookeeper is throwing
an error after that. Mostly it is throwing a "-5" error code which is a
marshalling/unmarshalling error. I'm using the perl interface of the
zookeeper.

My question is: Is there a limit to the maximum number of ZNODES a
zookeeper instance can hold or this is limited by the system memory?

Any pointers on how to avoid the error would be very helpful.

Thanks,
Aniket Chakrabarti
PhD student
The Ohio State University

On Wed, Nov 9, 2011 at 1:14 PM, Aniket Chakrabarti
<[EMAIL PROTECTED]> wrote:
> I am trying to load a huge matrix(100,000 by 500) to my zookeeper instance.
> Each element of the matrix is a ZNODE and value of each element is a
> digit(0-9).
>
> But I'm only able to load around 1000 x 500 nodes. Zookeeper is throwing an
> error after that. Mostly it is throwing a "-5" error code which is a
> marshalling/unmarshalling error. I'm using the perl interface of the
> zookeeper.
>
> My question is: Is there a limit to the maximum number of ZNODES a zookeeper
> instance can hold or this is limited by the system memory?
>
> Any pointers on how to avoid the error would be very helpful.

Available heap memory is really the only limit. Try

$ JVMFLAGS=-D-Xmx<mem in gig>g bin/zkServer.sh

also

$ sudo jmap -heap <jvm pid>

will give you some insight into whether it was set correctly or not
(ie MaxHeapSize)

The most I've tried is 5 million znodes with 25 million watches (using
zkpython, but zkperl should be fine). iirc that was 8gig heap, but
ymmv depending on data size.

You may also need to tune the GC at some point (would suggest turning
on cms and parallel collector) to limit stop the world pauses.

Regards,

Patrick

er, make that JVMFLAGS=-Xmx<mem in gig>g bin/zkServer.sh

(no -D)

Patrick

On Wed, Nov 9, 2011 at 2:31 PM, Patrick Hunt <[EMAIL PROTECTED]> wrote:
> On Wed, Nov 9, 2011 at 1:14 PM, Aniket Chakrabarti
> <[EMAIL PROTECTED]> wrote:
>> I am trying to load a huge matrix(100,000 by 500) to my zookeeper instance.
>> Each element of the matrix is a ZNODE and value of each element is a
>> digit(0-9).
>>
>> But I'm only able to load around 1000 x 500 nodes. Zookeeper is throwing an
>> error after that. Mostly it is throwing a "-5" error code which is a
>> marshalling/unmarshalling error. I'm using the perl interface of the
>> zookeeper.
>>
>> My question is: Is there a limit to the maximum number of ZNODES a zookeeper
>> instance can hold or this is limited by the system memory?
>>
>> Any pointers on how to avoid the error would be very helpful.
>
> Available heap memory is really the only limit. Try
>
> $ JVMFLAGS=-D-Xmx<mem in gig>g bin/zkServer.sh
>
> also
>
> $ sudo jmap -heap <jvm pid>
>
> will give you some insight into whether it was set correctly or not
> (ie MaxHeapSize)
>
> The most I've tried is 5 million znodes with 25 million watches (using
> zkpython, but zkperl should be fine). iirc that was 8gig heap, but
> ymmv depending on data size.
>
> You may also need to tune the GC at some point (would suggest turning
> on cms and parallel collector) to limit stop the world pauses.
>
> Regards,
>
> Patrick
>

Hi Patrick,

I tried setting the max heap size to 2G, but still after creating 1027 x
480 nodes zookeeper is throwing the error code "-5". Following is the
exception I'm seeing:

-----------------------------------------------------------------------------------------------------------------------------------------------
2011-11-09 23:42:49,484 - ERROR
[ProcessThread:-1:PrepRequestProcessor@415] - Failed to process
sessionid:0x1338bb71d340000 type:create cxid:0x4ec3599e
zxid:0xfffffffffffffffe txntype:unknown reqpath:/4/158:1096
java.nio.BufferOverflowException
         at java.nio.charset.CoderResult.throwException(Unknown Source)
         at java.lang.StringCoding$StringDecoder.decode(Unknown Source)
         at java.lang.StringCoding.decode(Unknown Source)
         at java.lang.String.<init>(Unknown Source)
         at java.lang.String.<init>(Unknown Source)
         at
org.apache.jute.BinaryInputArchive.readString(BinaryInputArchive.java:83)
         at org.apache.zookeeper.data.Id.deserialize(Id.java:55)
         at
org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
         at org.apache.zookeeper.data.ACL.deserialize(ACL.java:57)
         at
org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
         at
org.apache.zookeeper.proto.CreateRequest.deserialize(CreateRequest.java:92)
         at
org.apache.zookeeper.server.ZooKeeperServer.byteBuffer2Record(ZooKeeperServer.java:599)
         at
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:216)
         at
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114)
2011-11-09 23:42:49,485 - ERROR
[ProcessThread:-1:PrepRequestProcessor@428] - Dumping request buffer:
0x000b2f342f3135383a3130393600013100010001f0005776f726c640006616e796f6e650000
-----------------------------------------------------------------------------------------------------------------------------

Please advise.

Thanks,
Aniket

On 11/9/2011 6:00 PM, Patrick Hunt wrote:
> er, make that JVMFLAGS=-Xmx<mem in gig>g bin/zkServer.sh
>
> (no -D)
>
> Patrick
>
> On Wed, Nov 9, 2011 at 2:31 PM, Patrick Hunt<[EMAIL PROTECTED]>  wrote:
>> On Wed, Nov 9, 2011 at 1:14 PM, Aniket Chakrabarti
>> <[EMAIL PROTECTED]>  wrote:
>>> I am trying to load a huge matrix(100,000 by 500) to my zookeeper instance.
>>> Each element of the matrix is a ZNODE and value of each element is a
>>> digit(0-9).
>>>
>>> But I'm only able to load around 1000 x 500 nodes. Zookeeper is throwing an
>>> error after that. Mostly it is throwing a "-5" error code which is a
>>> marshalling/unmarshalling error. I'm using the perl interface of the
>>> zookeeper.
>>>
>>> My question is: Is there a limit to the maximum number of ZNODES a zookeeper
>>> instance can hold or this is limited by the system memory?
>>>
>>> Any pointers on how to avoid the error would be very helpful.
>> Available heap memory is really the only limit. Try
>>
>> $ JVMFLAGS=-D-Xmx<mem in gig>g bin/zkServer.sh
>>
>> also
>>
>> $ sudo jmap -heap<jvm pid>
>>
>> will give you some insight into whether it was set correctly or not
>> (ie MaxHeapSize)
>>
>> The most I've tried is 5 million znodes with 25 million watches (using
>> zkpython, but zkperl should be fine). iirc that was 8gig heap, but
>> ymmv depending on data size.
>>
>> You may also need to tune the GC at some point (would suggest turning
>> on cms and parallel collector) to limit stop the world pauses.
>>
>> Regards,
>>
>> Patrick
>>

10.11.11 06:53, Aniket Chakrabarti написав(ла):
> Hi Patrick,
>
> I tried setting the max heap size to 2G, but still after creating 1027
> x 480 nodes zookeeper is throwing the error code "-5". Following is
> the exception I'm seeing:
>
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> 2011-11-09 23:42:49,484 - ERROR
> [ProcessThread:-1:PrepRequestProcessor@415] - Failed to process
> sessionid:0x1338bb71d340000 type:create cxid:0x4ec3599e
> zxid:0xfffffffffffffffe txntype:unknown reqpath:/4/158:1096
> java.nio.BufferOverflowException
> at java.nio.charset.CoderResult.throwException(Unknown Source)
> at java.lang.StringCoding$StringDecoder.decode(Unknown Source)
> at java.lang.StringCoding.decode(Unknown Source)
> at java.lang.String.<init>(Unknown Source)
> at java.lang.String.<init>(Unknown Source)
> at
> org.apache.jute.BinaryInputArchive.readString(BinaryInputArchive.java:83)
> at org.apache.zookeeper.data.Id.deserialize(Id.java:55)
> at
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at org.apache.zookeeper.data.ACL.deserialize(ACL.java:57)
> at
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at
> org.apache.zookeeper.proto.CreateRequest.deserialize(CreateRequest.java:92)
> at
> org.apache.zookeeper.server.ZooKeeperServer.byteBuffer2Record(ZooKeeperServer.java:599)
> at
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:216)
> at
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114)
> 2011-11-09 23:42:49,485 - ERROR
> [ProcessThread:-1:PrepRequestProcessor@428] - Dumping request buffer:
> 0x000b2f342f3135383a3130393600013100010001f0005776f726c640006616e796f6e650000
> -----------------------------------------------------------------------------------------------------------------------------
>
>
> Please advise.

There is a limit on how many nodes can single parent node have if you
use list operation. The limit is not direct, but rather it's a limit on
response size. It can be changed (don't remember if with a setting or
compile time).

Best regards, Vitalii Tymchyshyn

On Thu, Nov 10, 2011 at 2:48 AM, Vitalii Tymchyshyn <[EMAIL PROTECTED]> wrote:
> 10.11.11 06:53, Aniket Chakrabarti написав(ла):
> There is a limit on how many nodes can single parent node have if you use
> list operation. The limit is not direct, but rather it's a limit on response
> size. It can be changed (don't remember if with a setting or compile time).

Good point. jute.maxbuffer, see the admin guide. You can set it at
runtime (client side).

https://issues.apache.org/jira/browse/ZOOKEEPER-1162

Patrick

Each node can take a fair bit of overhead in memory.   That 2G you
allocated may not go as far as you might like.  I am a bit surprised if
this eats as much memory as your test seems to imply, but it isn't
impossible.

Have you tried a row or column compressed format?  Do you really need to
store elements separately?

Also, Zookeeper is intended primarily as a coordination service.  That
tends to allow/encourage design decisions that may have a negative impact
on using it as a data store.  The fact is, it tends to do much better than
you might expect at datastore tasks, but that doesn't change the fact that
it really isn't a great platform for that.

On Wed, Nov 9, 2011 at 4:14 PM, Aniket Chakrabarti <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am trying to load a huge matrix(100,000 by 500) to my zookeeper
> instance. Each element of the matrix is a ZNODE and value of each element
> is a digit(0-9).
>
> But I'm only able to load around 1000 x 500 nodes. Zookeeper is throwing
> an error after that. Mostly it is throwing a "-5" error code which is a
> marshalling/unmarshalling error. I'm using the perl interface of the
> zookeeper.
>
> My question is: Is there a limit to the maximum number of ZNODES a
> zookeeper instance can hold or this is limited by the system memory?
>
> Any pointers on how to avoid the error would be very helpful.
>
> Thanks,
> Aniket Chakrabarti
> PhD student
> The Ohio State University
>

Good points Ted.

Aniket, you might try using visualvm or something similar to monitor
heap usage during your test:
http://visualvm.java.net/
It will give you some insight into how quickly the heap is being used up.

You're reporting approx 4k/znode. That seems way high to me. Are you
sure that the data for each znode is small? (you initially reported it
being a digit).

I just ran a quick test using yourkit and I'm seeing more like 400
bytes per znode.

You might start bin/zkCli.sh and use the "stat" command to get more
insight into your znodes (for example the length).

Patrick

On Wed, Nov 9, 2011 at 9:07 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:
> Each node can take a fair bit of overhead in memory.   That 2G you
> allocated may not go as far as you might like.  I am a bit surprised if
> this eats as much memory as your test seems to imply, but it isn't
> impossible.
>
> Have you tried a row or column compressed format?  Do you really need to
> store elements separately?
>
> Also, Zookeeper is intended primarily as a coordination service.  That
> tends to allow/encourage design decisions that may have a negative impact
> on using it as a data store.  The fact is, it tends to do much better than
> you might expect at datastore tasks, but that doesn't change the fact that
> it really isn't a great platform for that.
>
> On Wed, Nov 9, 2011 at 4:14 PM, Aniket Chakrabarti <
> [EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> I am trying to load a huge matrix(100,000 by 500) to my zookeeper
>> instance. Each element of the matrix is a ZNODE and value of each element
>> is a digit(0-9).
>>
>> But I'm only able to load around 1000 x 500 nodes. Zookeeper is throwing
>> an error after that. Mostly it is throwing a "-5" error code which is a
>> marshalling/unmarshalling error. I'm using the perl interface of the
>> zookeeper.
>>
>> My question is: Is there a limit to the maximum number of ZNODES a
>> zookeeper instance can hold or this is limited by the system memory?
>>
>> Any pointers on how to avoid the error would be very helpful.
>>
>> Thanks,
>> Aniket Chakrabarti
>> PhD student
>> The Ohio State University
>>
>