Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> Zookeeper ensemble backup questions?

Copy link to this message
Re: Zookeeper ensemble backup questions?
You don't need to stop ZooKeeper or wait until the system become idle.
You can do a live backup.

When you perform a backup, and restore from that backup.  This will give
you the same result as if ZooKeeper has crashed and restart at some point
during backup is being performed.

Again, if you need to make a backup of a very specific point in time, you
can either stop zookeeper or block all client port.   Or write some tool
to process txnlog if stopping zookeeper is not possible.
Thawan Kooburat

On 7/19/13 3:32 PM, "jack ma" <[EMAIL PROTECTED]> wrote:

>Thanks. Yes, we could just backup latest snapshot and transaction log. But
>how to prevent those files are not updated by zookeeper during our backup
>procedure. You mentioned to put the server to idle. What does that mean?
>put whole zookeeper ensemble to idle or just the server we backed up from?
>What is the zookeeper command for that?
>Appreciate for your help.
>On Fri, Jul 19, 2013 at 2:24 PM, Thawan Kooburat <[EMAIL PROTECTED]> wrote:
>> If you back up the entire data/snap dir at any given time on one of the
>> machine.  You can restore the system using the backup.  It will include
>> all the committed txn and some inf-light txn up to that point
>> If you need to be able to optimize the size of backup then you will have
>> to understand how the server load the snapshot and txnlog. In short, you
>> don't need the whole dir but just the most recently files
>> If you really need to be specific about which txns are included in the
>> backup, one easy way is to make the system idle before doing backup.
>> Otherwise, you need to write a tool to process the txnlog and snapshot
>> file.
>> So there is nothing prevent you from doing a zookeeper backup on the
>> current release (although it might not be convenient and require some
>> knowledge about zookeeper)
>> --
>> Thawan Kooburat
>> On 7/19/13 11:42 AM, "jack ma" <[EMAIL PROTECTED]> wrote:
>> >Thanks Sergei.
>> >
>> >That is great improvement idea for the zookeeper. I think that
>> >is
>> >planning to add a new 4lrt command "snap", but it is not ready yet.
>> >
>> >My original questions is based on the current version of zookeeper
>> >(3.4.5),
>> >do you have any answers for them?
>> >
>> >Appreciate for the help.
>> >
>> >thanks
>> >Jack
>> >
>> >
>> >
>> >
>> >On Fri, Jul 19, 2013 at 11:19 AM, Sergey Maslyakov
>> ><[EMAIL PROTECTED]>wrote:
>> >
>> >> Jack,
>> >>
>> >> Here is how I see the backup process happening.
>> >>
>> >> 1. Zookeeper server can be changed to support a new 4lw that will
>> >>out
>> >> the current state of the DataTree into a snapshot file with the path
>> >> name provided as an argument to this new command (barring all the
>> >> permissions, disk space, and other system-level restrictions).
>> >>Probably, I
>> >> would ask Zookeeper to save the snapshot in a directory outside of
>> >> standard "dataLog" for the sake of cleanliness.
>> >>
>> >> 2. When Zookeeper server responds to the new "snapshot" command with
>> >> success indication, the requesting process knows that the file has
>> >> written out and it can go and process it. It can add some metadata
>> >> create an archive to store it somewhere, for example. Alternatively,
>> >> Zookeeper server could stream the data it would have written into a
>> >> snapshot as the response to the new "snapshot" command. This way, the
>> >> client becomes responsible for persistence and this lifts a number of
>> >> permission-related issues (but raises some other issues too). Oh,
>>and by
>> >> the way, it looks like snapshot files are rather compressible. I did
>> >> the factor of 20 and more on the data that I have.
>> >>
>> >> 3. Disk cleanups are performed.
>> >>
>> >> With this backup procedure the restore would turn into:
>> >>
>> >> 1. Stopping all ensemble mebers
>> >>
>> >> 2. Wiping out dataDir/version-2 and dataLogDir/version-2
>> >>
>> >> 3. Restoring the snapshot taken by the above backup procedure on one