Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper, mail # user - Zookeeper ensemble backup questions?


Copy link to this message
-
Re: Zookeeper ensemble backup questions?
jack ma 2013-07-19, 22:28
Thanks Sergey.

That is great. Did you contribute you work back to zookeeper? When you take
a snapshot, did you have to block the zookeeper to accept for request?
On Fri, Jul 19, 2013 at 12:29 PM, Sergey Maslyakov <[EMAIL PROTECTED]>wrote:

> A word of preemptive self-defense: I am not an experienced Java developer.
> Please, don't throw rotten eggs at me if I did not follow well-known Java
> coding patterns :)
>
>
> Regards,
> /Sergey
>
>
> On Fri, Jul 19, 2013 at 2:15 PM, Sergey Maslyakov <[EMAIL PROTECTED]>
> wrote:
>
> > I can share this patch based on 3.4.5, which does thee trick.
> >
> > It adds a "snps" 4lw command that accepts one mandatory argument, which
> is
> > an absolute path for the direcotry where the snapshot file will be
> dropped.
> > The "absoluteness" of the path s verified by UNIX rules. Not sure how it
> > would work in Windows, though. The target directory must exist and be
> > writeable by the effective UID of Zookeeper server.
> >
> > If the operation was successful, Zookeeper server responds back with the
> > absolute path of the snapshot file. You can watch for the '/' character
> to
> > trigger your reaction to the response.
> >
> > In my case, a 700MB snapshot takes about 30 seconds to write out.
> >
> > Please see several examples below:
> >
> > ~ $ mkdir /tmp/snapshot-test
> >
> > ~ $ telnet localhost 12181
> > Trying 127.0.0.1...
> > Connected to localhost.
> > Escape character is '^]'.
> > snps /tmp/snapshot-test
> > /tmp/snapshot-test/snapshot.316c8
> > Connection to localhost closed by foreign host.
> >
> > ~ $ ls -al /tmp/snapshot-test/snapshot.316c8
> > -rw-r--r--   1 srvr     srvr     719602373 Jul 19 14:09
> > /tmp/snapshot-test/snapshot.316c8
> >
> > ~ $ telnet localhost 12181
> > Trying 127.0.0.1...
> > Connected to localhost.
> > Escape character is '^]'.
> > snps blah
> > Snapshot directory path must be absoulte, i.e., it must start with '/'.
> > Path "blah" does not meet the criteria.
> > Connection to localhost closed by foreign host.
> >
> > ~ $ telnet localhost 12181
> > Trying 127.0.0.1...
> > Connected to localhost.
> > Escape character is '^]'.
> > snps /tmp/blah
> > Error while serializing snapshot into /tmp/blah/snapshot.316c8.
> > /tmp/blah/snapshot.316c8 (No such file or directory)
> > Connection to localhost closed by foreign host.
> >
> > ~ $ telnet localhost 12181
> > Trying 127.0.0.1...
> > Connected to localhost.
> > Escape character is '^]'.
> > snps
> > Snapshot directory path must be absoulte, i.e., it must start with '/'.
> > Path "" does not meet the criteria.
> > Connection to localhost closed by foreign host.
> >
> > ~ $
> >
> >
> >
> >
> > On Fri, Jul 19, 2013 at 1:42 PM, jack ma <[EMAIL PROTECTED]> wrote:
> >
> >> Thanks Sergei.
> >>
> >> That is great improvement idea for the zookeeper. I think that zookeeper
> >> is
> >> planning to add a new 4lrt command "snap", but it is not ready yet.
> >>
> >> My original questions is based on the current version of zookeeper
> >> (3.4.5),
> >> do you have any answers for them?
> >>
> >> Appreciate for the help.
> >>
> >> thanks
> >> Jack
> >>
> >>
> >>
> >>
> >> On Fri, Jul 19, 2013 at 11:19 AM, Sergey Maslyakov <[EMAIL PROTECTED]
> >> >wrote:
> >>
> >> > Jack,
> >> >
> >> > Here is how I see the backup process happening.
> >> >
> >> > 1. Zookeeper server can be changed to support a new 4lw that will
> write
> >> out
> >> > the current state of the DataTree into a snapshot file with the path
> and
> >> > name provided as an argument to this new command (barring all the
> >> > permissions, disk space, and other system-level restrictions).
> >> Probably, I
> >> > would ask Zookeeper to save the snapshot in a directory outside of the
> >> > standard "dataLog" for the sake of cleanliness.
> >> >
> >> > 2. When Zookeeper server responds to the new "snapshot" command with
> >> > success indication, the requesting process knows that the file has
> been
> >> > written out and it can go and process it. It can add some metadata and