Is zookeeper responsible for the backup/replication of -ROOT- and .META.
files? It looks like I need multiple HBase masters setup to achieve high
availability. In the multiple masters setup, would there be any data loss in
the switch over after the first master became unavailable.

Is zookeeper responsible for the backup/replication of ROOT and META files?
It looks like I need multiple HBase masters setup to achieve high
availability. In the multiple masters setup, would there be any data loss in
the switch over after the first master became unavailable.

Thank you

On Fri, Jan 28, 2011 at 8:10 AM, Weishung Chung <[EMAIL PROTECTED]> wrote:
> Is zookeeper responsible for the backup/replication of -ROOT- and .META.
> files?

No.  These are kept in HDFS and rely on its replication.

> It looks like I need multiple HBase masters setup to achieve high
> availability. In the multiple masters setup, would there be any data loss in
> the switch over after the first master became unavailable.
>

No.  Master is not in the read/write path.  Cluster can continue
responding to read/writes even when Master(s) is (are) down.
St.Ack

Correct me if I am wrong :)
In HConnectionManager, it seems to me that a zookeeper instance is used to
get to the HBase master for META and ROOT info. What would happen if HBase
master became unavailable? Would zookeeper be able to get the ROOT and META
info from another backup/replicated master? Sorry because I haven't got a
chance to browse deeper in the zookeeper codes yet :(

On Fri, Jan 28, 2011 at 10:35 AM, Stack <[EMAIL PROTECTED]> wrote:

> On Fri, Jan 28, 2011 at 8:10 AM, Weishung Chung <[EMAIL PROTECTED]>
> wrote:
> > Is zookeeper responsible for the backup/replication of -ROOT- and .META.
> > files?
>
> No.  These are kept in HDFS and rely on its replication.
>
> > It looks like I need multiple HBase masters setup to achieve high
> > availability. In the multiple masters setup, would there be any data loss
> in
> > the switch over after the first master became unavailable.
> >
>
> No.  Master is not in the read/write path.  Cluster can continue
> responding to read/writes even when Master(s) is (are) down.
> St.Ack
>

On Fri, Jan 28, 2011 at 8:52 AM, Weishung Chung <[EMAIL PROTECTED]> wrote:
> Correct me if I am wrong :)
> In HConnectionManager, it seems to me that a zookeeper instance is used to
> get to the HBase master for META and ROOT info. What would happen if HBase
> master became unavailable? Would zookeeper be able to get the ROOT and META
> info from another backup/replicated master? Sorry because I haven't got a
> chance to browse deeper in the zookeeper codes yet :(
>

The locations of root is kept in zk and that of meta in the root
region, not in master.  If master goes away, cluster continues to run.
St.Ack

Great, thank you :D
I guess I need to read up more on zookeeper.

On Fri, Jan 28, 2011 at 10:56 AM, Stack <[EMAIL PROTECTED]> wrote:

> On Fri, Jan 28, 2011 at 8:52 AM, Weishung Chung <[EMAIL PROTECTED]>
> wrote:
> > Correct me if I am wrong :)
> > In HConnectionManager, it seems to me that a zookeeper instance is used
> to
> > get to the HBase master for META and ROOT info. What would happen if
> HBase
> > master became unavailable? Would zookeeper be able to get the ROOT and
> META
> > info from another backup/replicated master? Sorry because I haven't got a
> > chance to browse deeper in the zookeeper codes yet :(
> >
>
> The locations of root is kept in zk and that of meta in the root
> region, not in master.  If master goes away, cluster continues to run.
> St.Ack
>

I also don't have a solid understanding of the responsibilities of
master, but it seems like it's job is really about managing regions
(i.e., coordinating splits and compactions, etc.) and updating ROOT
and META. Is that correct?
On Fri, Jan 28, 2011 at 9:31 AM, Weishung Chung <[EMAIL PROTECTED]> wrote:
> Great, thank you :D
> I guess I need to read up more on zookeeper.
>
> On Fri, Jan 28, 2011 at 10:56 AM, Stack <[EMAIL PROTECTED]> wrote:
>
>> On Fri, Jan 28, 2011 at 8:52 AM, Weishung Chung <[EMAIL PROTECTED]>
>> wrote:
>> > Correct me if I am wrong :)
>> > In HConnectionManager, it seems to me that a zookeeper instance is used
>> to
>> > get to the HBase master for META and ROOT info. What would happen if
>> HBase
>> > master became unavailable? Would zookeeper be able to get the ROOT and
>> META
>> > info from another backup/replicated master? Sorry because I haven't got a
>> > chance to browse deeper in the zookeeper codes yet :(
>> >
>>
>> The locations of root is kept in zk and that of meta in the root
>> region, not in master.  If master goes away, cluster continues to run.
>> St.Ack
>>
>

On Fri, Jan 28, 2011 at 1:15 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
> I also don't have a solid understanding of the responsibilities of
> master, but it seems like it's job is really about managing regions
> (i.e., coordinating splits and compactions, etc.) and updating ROOT
> and META. Is that correct?
>
>

Yes.  It hosts the balancer and does bootsrapping on cluster startup
doing bulk initial assign.  On server crash, it runs the recovery
splitting WAL logs and getting regions back on line again.

It does not run splits.  That is done by the regionservers themselves.
 Regionservers inform master of the split when done so it can take
account of new state when running balancer.

We should do a write up on this.  Let me put this on the doc queue.

St.Ack

Thanks Stack, this is really helpful.

On Fri, Jan 28, 2011 at 2:06 PM, Stack <[EMAIL PROTECTED]> wrote:
> On Fri, Jan 28, 2011 at 1:15 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>> I also don't have a solid understanding of the responsibilities of
>> master, but it seems like it's job is really about managing regions
>> (i.e., coordinating splits and compactions, etc.) and updating ROOT
>> and META. Is that correct?
>>
>>
>
> Yes.  It hosts the balancer and does bootsrapping on cluster startup
> doing bulk initial assign.  On server crash, it runs the recovery
> splitting WAL logs and getting regions back on line again.
>
> It does not run splits.  That is done by the regionservers themselves.
>  Regionservers inform master of the split when done so it can take
> account of new state when running balancer.
>
> We should do a write up on this.  Let me put this on the doc queue.
>
> St.Ack
>