Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> Zookeeper ensemble backup questions?


Copy link to this message
-
Re: Zookeeper ensemble backup questions?
Thanks Sergey.

That is great. Did you contribute you work back to zookeeper? When you take
a snapshot, did you have to block the zookeeper to accept for request?
On Fri, Jul 19, 2013 at 12:29 PM, Sergey Maslyakov <[EMAIL PROTECTED]>wrote:

> A word of preemptive self-defense: I am not an experienced Java developer.
> Please, don't throw rotten eggs at me if I did not follow well-known Java
> coding patterns :)
>
>
> Regards,
> /Sergey
>
>
> On Fri, Jul 19, 2013 at 2:15 PM, Sergey Maslyakov <[EMAIL PROTECTED]>
> wrote:
>
> > I can share this patch based on 3.4.5, which does thee trick.
> >
> > It adds a "snps" 4lw command that accepts one mandatory argument, which
> is
> > an absolute path for the direcotry where the snapshot file will be
> dropped.
> > The "absoluteness" of the path s verified by UNIX rules. Not sure how it
> > would work in Windows, though. The target directory must exist and be
> > writeable by the effective UID of Zookeeper server.
> >
> > If the operation was successful, Zookeeper server responds back with the
> > absolute path of the snapshot file. You can watch for the '/' character
> to
> > trigger your reaction to the response.
> >
> > In my case, a 700MB snapshot takes about 30 seconds to write out.
> >
> > Please see several examples below:
> >
> > ~ $ mkdir /tmp/snapshot-test
> >
> > ~ $ telnet localhost 12181
> > Trying 127.0.0.1...
> > Connected to localhost.
> > Escape character is '^]'.
> > snps /tmp/snapshot-test
> > /tmp/snapshot-test/snapshot.316c8
> > Connection to localhost closed by foreign host.
> >
> > ~ $ ls -al /tmp/snapshot-test/snapshot.316c8
> > -rw-r--r--   1 srvr     srvr     719602373 Jul 19 14:09
> > /tmp/snapshot-test/snapshot.316c8
> >
> > ~ $ telnet localhost 12181
> > Trying 127.0.0.1...
> > Connected to localhost.
> > Escape character is '^]'.
> > snps blah
> > Snapshot directory path must be absoulte, i.e., it must start with '/'.
> > Path "blah" does not meet the criteria.
> > Connection to localhost closed by foreign host.
> >
> > ~ $ telnet localhost 12181
> > Trying 127.0.0.1...
> > Connected to localhost.
> > Escape character is '^]'.
> > snps /tmp/blah
> > Error while serializing snapshot into /tmp/blah/snapshot.316c8.
> > /tmp/blah/snapshot.316c8 (No such file or directory)
> > Connection to localhost closed by foreign host.
> >
> > ~ $ telnet localhost 12181
> > Trying 127.0.0.1...
> > Connected to localhost.
> > Escape character is '^]'.
> > snps
> > Snapshot directory path must be absoulte, i.e., it must start with '/'.
> > Path "" does not meet the criteria.
> > Connection to localhost closed by foreign host.
> >
> > ~ $
> >
> >
> >
> >
> > On Fri, Jul 19, 2013 at 1:42 PM, jack ma <[EMAIL PROTECTED]> wrote:
> >
> >> Thanks Sergei.
> >>
> >> That is great improvement idea for the zookeeper. I think that zookeeper
> >> is
> >> planning to add a new 4lrt command "snap", but it is not ready yet.
> >>
> >> My original questions is based on the current version of zookeeper
> >> (3.4.5),
> >> do you have any answers for them?
> >>
> >> Appreciate for the help.
> >>
> >> thanks
> >> Jack
> >>
> >>
> >>
> >>
> >> On Fri, Jul 19, 2013 at 11:19 AM, Sergey Maslyakov <[EMAIL PROTECTED]
> >> >wrote:
> >>
> >> > Jack,
> >> >
> >> > Here is how I see the backup process happening.
> >> >
> >> > 1. Zookeeper server can be changed to support a new 4lw that will
> write
> >> out
> >> > the current state of the DataTree into a snapshot file with the path
> and
> >> > name provided as an argument to this new command (barring all the
> >> > permissions, disk space, and other system-level restrictions).
> >> Probably, I
> >> > would ask Zookeeper to save the snapshot in a directory outside of the
> >> > standard "dataLog" for the sake of cleanliness.
> >> >
> >> > 2. When Zookeeper server responds to the new "snapshot" command with
> >> > success indication, the requesting process knows that the file has
> been
> >> > written out and it can go and process it. It can add some metadata and
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB