We have a situation where 99.9% of all data stored in zookeeper will be well
under the 1mb limit (probably under 1k as well) but there is a small
possibility that at some point users may do something to cross that
barrier.  I'd like to raise the max to a higher number realizing that if we
do hit that case, performance may suffer but it's better than having the app
crash.  i've looked through the docs and code and tried changing the
org.apache.zookeeper.server.quorum.QuorumCnxManager.PACKETMAXSIZE to a
larger number but something still seems to be blocking my test of larger
data sizes.

Is there some configuration parameter I am missing or code change i can
make?  Or have people solved this another way?  My first inclination was to
split up larger data streams across multiple nodes but that seems to cause
lots of problems with watches and atomicity that i don't think are easily
solvable.

- will

I think you can achieve this by using the command line argument to the
server:

-Djute.maxbuffer=<bytes>

~Jared

On Thu, Aug 11, 2011 at 7:08 AM, Will Johnson
<[EMAIL PROTECTED]>wrote:

> We have a situation where 99.9% of all data stored in zookeeper will be
> well
> under the 1mb limit (probably under 1k as well) but there is a small
> possibility that at some point users may do something to cross that
> barrier.  I'd like to raise the max to a higher number realizing that if we
> do hit that case, performance may suffer but it's better than having the
> app
> crash.  i've looked through the docs and code and tried changing the
> org.apache.zookeeper.server.quorum.QuorumCnxManager.PACKETMAXSIZE to a
> larger number but something still seems to be blocking my test of larger
> data sizes.
>
> Is there some configuration parameter I am missing or code change i can
> make?  Or have people solved this another way?  My first inclination was to
> split up larger data streams across multiple nodes but that seems to cause
> lots of problems with watches and atomicity that i don't think are easily
> solvable.
>
> - will
>

Another way to solve this is to use the multi command.

The idea would be that you would upload multiple pieces of the large object
separately into different znodes (without using multi)

Then you would update a pointer node that has references to the pieces while
controlling for the version of the pieces (using a multi).

On Thu, Aug 11, 2011 at 6:08 AM, Will Johnson
<[EMAIL PROTECTED]>wrote:

> We have a situation where 99.9% of all data stored in zookeeper will be
> well
> under the 1mb limit (probably under 1k as well) but there is a small
> possibility that at some point users may do something to cross that
> barrier.  ...
> Is there some configuration parameter I am missing or code change i can
> make?  Or have people solved this another way?
>

Currently, the limit on the sum of the sizes of the updates in the multi
command is still 1MB.  You cannot commit 5 1MB nodes in a multi-op.

~Jared

On Thu, Aug 11, 2011 at 11:20 AM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Another way to solve this is to use the multi command.
>
> The idea would be that you would upload multiple pieces of the large object
> separately into different znodes (without using multi)
>
> Then you would update a pointer node that has references to the pieces
> while
> controlling for the version of the pieces (using a multi).
>
> On Thu, Aug 11, 2011 at 6:08 AM, Will Johnson
> <[EMAIL PROTECTED]>wrote:
>
> > We have a situation where 99.9% of all data stored in zookeeper will be
> > well
> > under the 1mb limit (probably under 1k as well) but there is a small
> > possibility that at some point users may do something to cross that
> > barrier.  ...
> > Is there some configuration parameter I am missing or code change i can
> > make?  Or have people solved this another way?
> >
>

Yes.

But you can commit a single coordination file with version checks on 5
pre-existing 1MB files.

On Thu, Aug 11, 2011 at 11:00 AM, Jared Cantwell
<[EMAIL PROTECTED]>wrote:

> Currently, the limit on the sum of the sizes of the updates in the multi
> command is still 1MB.  You cannot commit 5 1MB nodes in a multi-op.
>
> ~Jared
>
> On Thu, Aug 11, 2011 at 11:20 AM, Ted Dunning <[EMAIL PROTECTED]>
> wrote:
>
> > Another way to solve this is to use the multi command.
> >
> > The idea would be that you would upload multiple pieces of the large
> object
> > separately into different znodes (without using multi)
> >
> > Then you would update a pointer node that has references to the pieces
> > while
> > controlling for the version of the pieces (using a multi).
> >
> > On Thu, Aug 11, 2011 at 6:08 AM, Will Johnson
> > <[EMAIL PROTECTED]>wrote:
> >
> > > We have a situation where 99.9% of all data stored in zookeeper will be
> > > well
> > > under the 1mb limit (probably under 1k as well) but there is a small
> > > possibility that at some point users may do something to cross that
> > > barrier.  ...
> > > Is there some configuration parameter I am missing or code change i can
> > > make?  Or have people solved this another way?
> > >
> >
>

Ahh, I missed that in your message.  This is a clever use of multi.

~Jared

On Thu, Aug 11, 2011 at 2:16 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Yes.
>
> But you can commit a single coordination file with version checks on 5
> pre-existing 1MB files.
>
> On Thu, Aug 11, 2011 at 11:00 AM, Jared Cantwell
> <[EMAIL PROTECTED]>wrote:
>
> > Currently, the limit on the sum of the sizes of the updates in the multi
> > command is still 1MB.  You cannot commit 5 1MB nodes in a multi-op.
> >
> > ~Jared
> >
> > On Thu, Aug 11, 2011 at 11:20 AM, Ted Dunning <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Another way to solve this is to use the multi command.
> > >
> > > The idea would be that you would upload multiple pieces of the large
> > object
> > > separately into different znodes (without using multi)
> > >
> > > Then you would update a pointer node that has references to the pieces
> > > while
> > > controlling for the version of the pieces (using a multi).
> > >
> > > On Thu, Aug 11, 2011 at 6:08 AM, Will Johnson
> > > <[EMAIL PROTECTED]>wrote:
> > >
> > > > We have a situation where 99.9% of all data stored in zookeeper will
> be
> > > > well
> > > > under the 1mb limit (probably under 1k as well) but there is a small
> > > > possibility that at some point users may do something to cross that
> > > > barrier.  ...
> > > > Is there some configuration parameter I am missing or code change i
> can
> > > > make?  Or have people solved this another way?
> > > >
> > >
> >
>