Hello,

I have been searching through the zookeeper documentation and the web for
information about reconfiguring a zookeeper ensemble. More precisely I am
interested in adding/removing zookeeper servers to/from a zookeeper
ensemble. Unfortunately, I could not find much information around,
therefore I am bringing up the issue on this list.

My questions are:

- Is there a safe procedure to add a zookeeper server to a zookeeper
ensemble?
- Is there a safe procedure to remove a zookeeper server from an existing
ensemble?

Some posts I have read suggest that these operations are possible. The
described procedure is to rewrite a new zookeeper configuration to all
involved servers and to restart them. However, given that I could not find
any information about how to carry out these two operations in zookeeper's
documentation makes me nervous about relying on the described procedure.

I would appreciate if anyone can point me to a different procedures to
carry out add/remove operations, or convince me that it is safe to use the
procedure described above. Safety for me is two-fold: zookeeper database
can not get corrupted (unusable), and state updates can not get lost.

--
Mohammad Abdul-Amir (Shamma)

See #6 in the FAQ: http://wiki.apache.org/hadoop/ZooKeeper/FAQ

Also, FYI, we recently open sourced a ZooKeeper supervisor app that makes
these kinds of tasks simpler:

https://github.com/Netflix/exhibitor

-Jordan

On 4/26/12 2:46 PM, "Mohammad Abdul-Amir (Shamma)"
<[EMAIL PROTECTED]> wrote:

>Hello,
>
>I have been searching through the zookeeper documentation and the web for
>information about reconfiguring a zookeeper ensemble. More precisely I am
>interested in adding/removing zookeeper servers to/from a zookeeper
>ensemble. Unfortunately, I could not find much information around,
>therefore I am bringing up the issue on this list.
>
>My questions are:
>
>- Is there a safe procedure to add a zookeeper server to a zookeeper
>ensemble?
>- Is there a safe procedure to remove a zookeeper server from an existing
>ensemble?
>
>Some posts I have read suggest that these operations are possible. The
>described procedure is to rewrite a new zookeeper configuration to all
>involved servers and to restart them. However, given that I could not find
>any information about how to carry out these two operations in zookeeper's
>documentation makes me nervous about relying on the described procedure.
>
>I would appreciate if anyone can point me to a different procedures to
>carry out add/remove operations, or convince me that it is safe to use the
>procedure described above. Safety for me is two-fold: zookeeper database
>can not get corrupted (unusable), and state updates can not get lost.
>
>--
>Mohammad Abdul-Amir (Shamma)

On Thu, Apr 26, 2012 at 2:48 PM, Jordan Zimmerman <[EMAIL PROTECTED]>wrote:

> See #6 in the FAQ: http://wiki.apache.org/hadoop/ZooKeeper/FAQ
>
>
Okay, it seems that a zookeeper "upgrade" is possible according to FAQ.

Does removing/decommissioning a zookeeper server fall under the notion of
"upgrade" though? Does "upgrade" apply to any zookeeper config update?
> Also, FYI, we recently open sourced a ZooKeeper supervisor app that makes
> these kinds of tasks simpler:
>
>        https://github.com/Netflix/exhibitor
>
> -Jordan
>
> On 4/26/12 2:46 PM, "Mohammad Abdul-Amir (Shamma)"
> <[EMAIL PROTECTED]> wrote:
>
> >Hello,
> >
> >I have been searching through the zookeeper documentation and the web for
> >information about reconfiguring a zookeeper ensemble. More precisely I am
> >interested in adding/removing zookeeper servers to/from a zookeeper
> >ensemble. Unfortunately, I could not find much information around,
> >therefore I am bringing up the issue on this list.
> >
> >My questions are:
> >
> >- Is there a safe procedure to add a zookeeper server to a zookeeper
> >ensemble?
> >- Is there a safe procedure to remove a zookeeper server from an existing
> >ensemble?
> >
> >Some posts I have read suggest that these operations are possible. The
> >described procedure is to rewrite a new zookeeper configuration to all
> >involved servers and to restart them. However, given that I could not find
> >any information about how to carry out these two operations in zookeeper's
> >documentation makes me nervous about relying on the described procedure.
> >
> >I would appreciate if anyone can point me to a different procedures to
> >carry out add/remove operations, or convince me that it is safe to use the
> >procedure described above. Safety for me is two-fold: zookeeper database
> >can not get corrupted (unusable), and state updates can not get lost.
> >
> >--
> >Mohammad Abdul-Amir (Shamma)
>
>
--
Mohammad Abdul-Amir (Shamma)

Hi Mohammad,

Support for adding/removing servers as well as dynamic changes to
other parameters will be added as part of ZOOKEEPER-107, hopefully in
the 3.5.0 release.

Thanks,
Alex

On Thu, Apr 26, 2012 at 3:55 PM, Mohammad Abdul-Amir (Shamma)
<[EMAIL PROTECTED]> wrote:
> On Thu, Apr 26, 2012 at 2:48 PM, Jordan Zimmerman <[EMAIL PROTECTED]>wrote:
>
>> See #6 in the FAQ: http://wiki.apache.org/hadoop/ZooKeeper/FAQ
>>
>>
> Okay, it seems that a zookeeper "upgrade" is possible according to FAQ.
>
> Does removing/decommissioning a zookeeper server fall under the notion of
> "upgrade" though? Does "upgrade" apply to any zookeeper config update?
>
>
>> Also, FYI, we recently open sourced a ZooKeeper supervisor app that makes
>> these kinds of tasks simpler:
>>
>>        https://github.com/Netflix/exhibitor
>>
>> -Jordan
>>
>> On 4/26/12 2:46 PM, "Mohammad Abdul-Amir (Shamma)"
>> <[EMAIL PROTECTED]> wrote:
>>
>> >Hello,
>> >
>> >I have been searching through the zookeeper documentation and the web for
>> >information about reconfiguring a zookeeper ensemble. More precisely I am
>> >interested in adding/removing zookeeper servers to/from a zookeeper
>> >ensemble. Unfortunately, I could not find much information around,
>> >therefore I am bringing up the issue on this list.
>> >
>> >My questions are:
>> >
>> >- Is there a safe procedure to add a zookeeper server to a zookeeper
>> >ensemble?
>> >- Is there a safe procedure to remove a zookeeper server from an existing
>> >ensemble?
>> >
>> >Some posts I have read suggest that these operations are possible. The
>> >described procedure is to rewrite a new zookeeper configuration to all
>> >involved servers and to restart them. However, given that I could not find
>> >any information about how to carry out these two operations in zookeeper's
>> >documentation makes me nervous about relying on the described procedure.
>> >
>> >I would appreciate if anyone can point me to a different procedures to
>> >carry out add/remove operations, or convince me that it is safe to use the
>> >procedure described above. Safety for me is two-fold: zookeeper database
>> >can not get corrupted (unusable), and state updates can not get lost.
>> >
>> >--
>> >Mohammad Abdul-Amir (Shamma)
>>
>>
>
>
> --
> Mohammad Abdul-Amir (Shamma)

Alex,

It sounds like you are suggesting that zookeeper keeper does not support
adding/removing servers non-dynamically. If so, this contradicts with the
FAQ #6 that Jordan Zimmerman pointed out.

On Sun, Apr 29, 2012 at 11:32 PM, Alexander Shraer <[EMAIL PROTECTED]>wrote:

> Hi Mohammad,
>
> Support for adding/removing servers as well as dynamic changes to
> other parameters will be added as part of ZOOKEEPER-107, hopefully in
> the 3.5.0 release.
>
> Thanks,
> Alex
>
> On Thu, Apr 26, 2012 at 3:55 PM, Mohammad Abdul-Amir (Shamma)
> <[EMAIL PROTECTED]> wrote:
> > On Thu, Apr 26, 2012 at 2:48 PM, Jordan Zimmerman <
> [EMAIL PROTECTED]>wrote:
> >
> >> See #6 in the FAQ: http://wiki.apache.org/hadoop/ZooKeeper/FAQ
> >>
> >>
> > Okay, it seems that a zookeeper "upgrade" is possible according to FAQ.
> >
> > Does removing/decommissioning a zookeeper server fall under the notion of
> > "upgrade" though? Does "upgrade" apply to any zookeeper config update?
> >
> >
> >> Also, FYI, we recently open sourced a ZooKeeper supervisor app that
> makes
> >> these kinds of tasks simpler:
> >>
> >>        https://github.com/Netflix/exhibitor
> >>
> >> -Jordan
> >>
> >> On 4/26/12 2:46 PM, "Mohammad Abdul-Amir (Shamma)"
> >> <[EMAIL PROTECTED]> wrote:
> >>
> >> >Hello,
> >> >
> >> >I have been searching through the zookeeper documentation and the web
> for
> >> >information about reconfiguring a zookeeper ensemble. More precisely I
> am
> >> >interested in adding/removing zookeeper servers to/from a zookeeper
> >> >ensemble. Unfortunately, I could not find much information around,
> >> >therefore I am bringing up the issue on this list.
> >> >
> >> >My questions are:
> >> >
> >> >- Is there a safe procedure to add a zookeeper server to a zookeeper
> >> >ensemble?
> >> >- Is there a safe procedure to remove a zookeeper server from an
> existing
> >> >ensemble?
> >> >
> >> >Some posts I have read suggest that these operations are possible. The
> >> >described procedure is to rewrite a new zookeeper configuration to all
> >> >involved servers and to restart them. However, given that I could not
> find
> >> >any information about how to carry out these two operations in
> zookeeper's
> >> >documentation makes me nervous about relying on the described
> procedure.
> >> >
> >> >I would appreciate if anyone can point me to a different procedures to
> >> >carry out add/remove operations, or convince me that it is safe to use
> the
> >> >procedure described above. Safety for me is two-fold: zookeeper
> database
> >> >can not get corrupted (unusable), and state updates can not get lost.
> >> >
> >> >--
> >> >Mohammad Abdul-Amir (Shamma)
> >>
> >>
> >
> >
> > --
> > Mohammad Abdul-Amir (Shamma)
>

--
Mohammad Abdul-Amir (Shamma)

Technically, ZK does not support it. However, there is a methodology for
reconfiguring and updated an ensemble. I believe ZK-107 will allow the
changes to be made without needing to restart/reconfigure the cluster.

-JZ

On 4/30/12 11:49 AM, "Mohammad Abdul-Amir (Shamma)"
<[EMAIL PROTECTED]> wrote:

>Alex,
>
>It sounds like you are suggesting that zookeeper keeper does not support
>adding/removing servers non-dynamically. If so, this contradicts with the
>FAQ #6 that Jordan Zimmerman pointed out.
>
>On Sun, Apr 29, 2012 at 11:32 PM, Alexander Shraer
><[EMAIL PROTECTED]>wrote:
>
>> Hi Mohammad,
>>
>> Support for adding/removing servers as well as dynamic changes to
>> other parameters will be added as part of ZOOKEEPER-107, hopefully in
>> the 3.5.0 release.
>>
>> Thanks,
>> Alex
>>
>> On Thu, Apr 26, 2012 at 3:55 PM, Mohammad Abdul-Amir (Shamma)
>> <[EMAIL PROTECTED]> wrote:
>> > On Thu, Apr 26, 2012 at 2:48 PM, Jordan Zimmerman <
>> [EMAIL PROTECTED]>wrote:
>> >
>> >> See #6 in the FAQ: http://wiki.apache.org/hadoop/ZooKeeper/FAQ
>> >>
>> >>
>> > Okay, it seems that a zookeeper "upgrade" is possible according to
>>FAQ.
>> >
>> > Does removing/decommissioning a zookeeper server fall under the
>>notion of
>> > "upgrade" though? Does "upgrade" apply to any zookeeper config update?
>> >
>> >
>> >> Also, FYI, we recently open sourced a ZooKeeper supervisor app that
>> makes
>> >> these kinds of tasks simpler:
>> >>
>> >>        https://github.com/Netflix/exhibitor
>> >>
>> >> -Jordan
>> >>
>> >> On 4/26/12 2:46 PM, "Mohammad Abdul-Amir (Shamma)"
>> >> <[EMAIL PROTECTED]> wrote:
>> >>
>> >> >Hello,
>> >> >
>> >> >I have been searching through the zookeeper documentation and the
>>web
>> for
>> >> >information about reconfiguring a zookeeper ensemble. More
>>precisely I
>> am
>> >> >interested in adding/removing zookeeper servers to/from a zookeeper
>> >> >ensemble. Unfortunately, I could not find much information around,
>> >> >therefore I am bringing up the issue on this list.
>> >> >
>> >> >My questions are:
>> >> >
>> >> >- Is there a safe procedure to add a zookeeper server to a zookeeper
>> >> >ensemble?
>> >> >- Is there a safe procedure to remove a zookeeper server from an
>> existing
>> >> >ensemble?
>> >> >
>> >> >Some posts I have read suggest that these operations are possible.
>>The
>> >> >described procedure is to rewrite a new zookeeper configuration to
>>all
>> >> >involved servers and to restart them. However, given that I could
>>not
>> find
>> >> >any information about how to carry out these two operations in
>> zookeeper's
>> >> >documentation makes me nervous about relying on the described
>> procedure.
>> >> >
>> >> >I would appreciate if anyone can point me to a different procedures
>>to
>> >> >carry out add/remove operations, or convince me that it is safe to
>>use
>> the
>> >> >procedure described above. Safety for me is two-fold: zookeeper
>> database
>> >> >can not get corrupted (unusable), and state updates can not get
>>lost.
>> >> >
>> >> >--
>> >> >Mohammad Abdul-Amir (Shamma)
>> >>
>> >>
>> >
>> >
>> > --
>> > Mohammad Abdul-Amir (Shamma)
>>
>
>
>
>--
>Mohammad Abdul-Amir (Shamma)

By "methodology", do I understand you mean it is a safe way of
reconfiguration? If that is the case, what makes it a safe way of
configuration, albeit not supported explicitly by zookeeper?

On Mon, Apr 30, 2012 at 11:54 AM, Jordan Zimmerman
<[EMAIL PROTECTED]>wrote:

> Technically, ZK does not support it. However, there is a methodology for
> reconfiguring and updated an ensemble. I believe ZK-107 will allow the
> changes to be made without needing to restart/reconfigure the cluster.
>
> -JZ
>
> On 4/30/12 11:49 AM, "Mohammad Abdul-Amir (Shamma)"
> <[EMAIL PROTECTED]> wrote:
>
> >Alex,
> >
> >It sounds like you are suggesting that zookeeper keeper does not support
> >adding/removing servers non-dynamically. If so, this contradicts with the
> >FAQ #6 that Jordan Zimmerman pointed out.
> >
> >On Sun, Apr 29, 2012 at 11:32 PM, Alexander Shraer
> ><[EMAIL PROTECTED]>wrote:
> >
> >> Hi Mohammad,
> >>
> >> Support for adding/removing servers as well as dynamic changes to
> >> other parameters will be added as part of ZOOKEEPER-107, hopefully in
> >> the 3.5.0 release.
> >>
> >> Thanks,
> >> Alex
> >>
> >> On Thu, Apr 26, 2012 at 3:55 PM, Mohammad Abdul-Amir (Shamma)
> >> <[EMAIL PROTECTED]> wrote:
> >> > On Thu, Apr 26, 2012 at 2:48 PM, Jordan Zimmerman <
> >> [EMAIL PROTECTED]>wrote:
> >> >
> >> >> See #6 in the FAQ: http://wiki.apache.org/hadoop/ZooKeeper/FAQ
> >> >>
> >> >>
> >> > Okay, it seems that a zookeeper "upgrade" is possible according to
> >>FAQ.
> >> >
> >> > Does removing/decommissioning a zookeeper server fall under the
> >>notion of
> >> > "upgrade" though? Does "upgrade" apply to any zookeeper config update?
> >> >
> >> >
> >> >> Also, FYI, we recently open sourced a ZooKeeper supervisor app that
> >> makes
> >> >> these kinds of tasks simpler:
> >> >>
> >> >>        https://github.com/Netflix/exhibitor
> >> >>
> >> >> -Jordan
> >> >>
> >> >> On 4/26/12 2:46 PM, "Mohammad Abdul-Amir (Shamma)"
> >> >> <[EMAIL PROTECTED]> wrote:
> >> >>
> >> >> >Hello,
> >> >> >
> >> >> >I have been searching through the zookeeper documentation and the
> >>web
> >> for
> >> >> >information about reconfiguring a zookeeper ensemble. More
> >>precisely I
> >> am
> >> >> >interested in adding/removing zookeeper servers to/from a zookeeper
> >> >> >ensemble. Unfortunately, I could not find much information around,
> >> >> >therefore I am bringing up the issue on this list.
> >> >> >
> >> >> >My questions are:
> >> >> >
> >> >> >- Is there a safe procedure to add a zookeeper server to a zookeeper
> >> >> >ensemble?
> >> >> >- Is there a safe procedure to remove a zookeeper server from an
> >> existing
> >> >> >ensemble?
> >> >> >
> >> >> >Some posts I have read suggest that these operations are possible.
> >>The
> >> >> >described procedure is to rewrite a new zookeeper configuration to
> >>all
> >> >> >involved servers and to restart them. However, given that I could
> >>not
> >> find
> >> >> >any information about how to carry out these two operations in
> >> zookeeper's
> >> >> >documentation makes me nervous about relying on the described
> >> procedure.
> >> >> >
> >> >> >I would appreciate if anyone can point me to a different procedures
> >>to
> >> >> >carry out add/remove operations, or convince me that it is safe to
> >>use
> >> the
> >> >> >procedure described above. Safety for me is two-fold: zookeeper
> >> database
> >> >> >can not get corrupted (unusable), and state updates can not get
> >>lost.
> >> >> >
> >> >> >--
> >> >> >Mohammad Abdul-Amir (Shamma)
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > Mohammad Abdul-Amir (Shamma)
> >>
> >
> >
> >
> >--
> >Mohammad Abdul-Amir (Shamma)
>
>
--
Mohammad Abdul-Amir (Shamma)

exactly - ZK-107 allows fully automatic reconfiguration without any
restarts (it adds this as an API command to zookeeper). We also have a
mechanism to rebalance clients across the new set of servers.

Currently you can do a rolling restart to change the configuration,
which is quite tricky to do correctly, but I believe that Jordan's
Exhibitor tool simplifies the process.

Alex

On Mon, Apr 30, 2012 at 11:54 AM, Jordan Zimmerman
<[EMAIL PROTECTED]> wrote:
> Technically, ZK does not support it. However, there is a methodology for
> reconfiguring and updated an ensemble. I believe ZK-107 will allow the
> changes to be made without needing to restart/reconfigure the cluster.
>
> -JZ
>
> On 4/30/12 11:49 AM, "Mohammad Abdul-Amir (Shamma)"
> <[EMAIL PROTECTED]> wrote:
>
>>Alex,
>>
>>It sounds like you are suggesting that zookeeper keeper does not support
>>adding/removing servers non-dynamically. If so, this contradicts with the
>>FAQ #6 that Jordan Zimmerman pointed out.
>>
>>On Sun, Apr 29, 2012 at 11:32 PM, Alexander Shraer
>><[EMAIL PROTECTED]>wrote:
>>
>>> Hi Mohammad,
>>>
>>> Support for adding/removing servers as well as dynamic changes to
>>> other parameters will be added as part of ZOOKEEPER-107, hopefully in
>>> the 3.5.0 release.
>>>
>>> Thanks,
>>> Alex
>>>
>>> On Thu, Apr 26, 2012 at 3:55 PM, Mohammad Abdul-Amir (Shamma)
>>> <[EMAIL PROTECTED]> wrote:
>>> > On Thu, Apr 26, 2012 at 2:48 PM, Jordan Zimmerman <
>>> [EMAIL PROTECTED]>wrote:
>>> >
>>> >> See #6 in the FAQ: http://wiki.apache.org/hadoop/ZooKeeper/FAQ
>>> >>
>>> >>
>>> > Okay, it seems that a zookeeper "upgrade" is possible according to
>>>FAQ.
>>> >
>>> > Does removing/decommissioning a zookeeper server fall under the
>>>notion of
>>> > "upgrade" though? Does "upgrade" apply to any zookeeper config update?
>>> >
>>> >
>>> >> Also, FYI, we recently open sourced a ZooKeeper supervisor app that
>>> makes
>>> >> these kinds of tasks simpler:
>>> >>
>>> >>        https://github.com/Netflix/exhibitor
>>> >>
>>> >> -Jordan
>>> >>
>>> >> On 4/26/12 2:46 PM, "Mohammad Abdul-Amir (Shamma)"
>>> >> <[EMAIL PROTECTED]> wrote:
>>> >>
>>> >> >Hello,
>>> >> >
>>> >> >I have been searching through the zookeeper documentation and the
>>>web
>>> for
>>> >> >information about reconfiguring a zookeeper ensemble. More
>>>precisely I
>>> am
>>> >> >interested in adding/removing zookeeper servers to/from a zookeeper
>>> >> >ensemble. Unfortunately, I could not find much information around,
>>> >> >therefore I am bringing up the issue on this list.
>>> >> >
>>> >> >My questions are:
>>> >> >
>>> >> >- Is there a safe procedure to add a zookeeper server to a zookeeper
>>> >> >ensemble?
>>> >> >- Is there a safe procedure to remove a zookeeper server from an
>>> existing
>>> >> >ensemble?
>>> >> >
>>> >> >Some posts I have read suggest that these operations are possible.
>>>The
>>> >> >described procedure is to rewrite a new zookeeper configuration to
>>>all
>>> >> >involved servers and to restart them. However, given that I could
>>>not
>>> find
>>> >> >any information about how to carry out these two operations in
>>> zookeeper's
>>> >> >documentation makes me nervous about relying on the described
>>> procedure.
>>> >> >
>>> >> >I would appreciate if anyone can point me to a different procedures
>>>to
>>> >> >carry out add/remove operations, or convince me that it is safe to
>>>use
>>> the
>>> >> >procedure described above. Safety for me is two-fold: zookeeper
>>> database
>>> >> >can not get corrupted (unusable), and state updates can not get
>>>lost.
>>> >> >
>>> >> >--
>>> >> >Mohammad Abdul-Amir (Shamma)
>>> >>
>>> >>
>>> >
>>> >
>>> > --
>>> > Mohammad Abdul-Amir (Shamma)
>>>
>>
>>
>>
>>--
>>Mohammad Abdul-Amir (Shamma)
>