hi all,
I am updating production cluster from 0.90.2 to 0.94 .  My table's size is about 20TB+ .
Scheduled update includes upgrading both hbase,hadoop version,
 and I am also changing user with which I  start up both hadoop and hbase cluster  from user "A" to user "B".

It's production environment , so I wanted know what steps I should not miss regarding this upgrade.
Table is kinda big and I don't have backup cluster to backup my data.
I wanted to know will there be a data loss scenario if I rollback after having failed to upgrade. thanks.

--
Best Regards
Yusup Ashrap

Hi,

We also upgraded the version very recently. If you can afford couple of
minutes downtime then you can safely bring down the cluster and do upgrade.
As such, there will be no data loss, but be careful with splits. The
default split policy has been changed in this version if I am not wrong. It
causes some weird things.

Thanks
Kiran
On Tue, Feb 26, 2013 at 1:03 PM, Yusup Ashrap <[EMAIL PROTECTED]> wrote:

> hi all,
> I am updating production cluster from 0.90.2 to 0.94 .  My table's size is
> about 20TB+ .
> Scheduled update includes upgrading both hbase,hadoop version,
>  and I am also changing user with which I  start up both hadoop and hbase
> cluster  from user "A" to user "B".
>
> It's production environment , so I wanted know what steps I should not
> miss regarding this upgrade.
> Table is kinda big and I don't have backup cluster to backup my data.
> I wanted to know will there be a data loss scenario if I rollback after
> having failed to upgrade. thanks.
>
>
>
>
>
> --
> Best Regards
> Yusup Ashrap
>
>
--
Thank you
Kiran Sarvabhotla

-----Even a correct decision is wrong when it is taken late

Hi Kiran , thanks for reply
>From what I've read from online docs , downtime is inevitable for upgrading from 0.90.2 to 0.94,
and I can afford some downtime. I cannot afford dataloss, so I am concerning potential problems with rolling back to 0.90.2 if I fail to upgrade.
--
Best Regards
Yusup Ashrap
On Tuesday, February 26, 2013 at 3:53 PM, kiran wrote:

> Hi,
>
> We also upgraded the version very recently. If you can afford couple of
> minutes downtime then you can safely bring down the cluster and do upgrade.
> As such, there will be no data loss, but be careful with splits. The
> default split policy has been changed in this version if I am not wrong. It
> causes some weird things.
>
> Thanks
> Kiran
>
>
> On Tue, Feb 26, 2013 at 1:03 PM, Yusup Ashrap <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>
> > hi all,
> > I am updating production cluster from 0.90.2 to 0.94 . My table's size is
> > about 20TB+ .
> > Scheduled update includes upgrading both hbase,hadoop version,
> > and I am also changing user with which I start up both hadoop and hbase
> > cluster from user "A" to user "B".
> >
> > It's production environment , so I wanted know what steps I should not
> > miss regarding this upgrade.
> > Table is kinda big and I don't have backup cluster to backup my data.
> > I wanted to know will there be a data loss scenario if I rollback after
> > having failed to upgrade. thanks.
> >
> >
> >
> >
> >
> > --
> > Best Regards
> > Yusup Ashrap
> >
>
>
>
> --
> Thank you
> Kiran Sarvabhotla
>
> -----Even a correct decision is wrong when it is taken late

Well if you can afford a longer downtime, you can always distcp your
existing hbase data. This way if things get screwed up you can always
restore a 0.90.x on that old backup. You cannot distcp while the cluster is
running since it will not be able to get locks on file (I think I faced
that issue but not sure since I did this upgrade mid-2012).

When I did an upgrade, I brought up a test 0.94 cluster using the distcp
backup (I took like 2-3 copies since I was very paranoid). Once I ran all
my production jobs against the test instance and I was happy with all the
results, that's when I re-ran the script to do the migration. It was all
scripted out but I no longer have access to those scripts or else I would
have shared it with you.

Thanks,
Viral

On Tue, Feb 26, 2013 at 12:00 AM, Yusup Ashrap <[EMAIL PROTECTED]> wrote:

> Hi Kiran , thanks for reply
> From what I've read from online docs , downtime is inevitable for
> upgrading from 0.90.2 to 0.94,
> and I can afford some downtime. I cannot afford dataloss, so I am
> concerning potential problems with rolling back to 0.90.2 if I fail to
> upgrade.
>
>
> --
> Best Regards
> Yusup Ashrap
>
>
> On Tuesday, February 26, 2013 at 3:53 PM, kiran wrote:
>
> > Hi,
> >
> > We also upgraded the version very recently. If you can afford couple of
> > minutes downtime then you can safely bring down the cluster and do
> upgrade.
> > As such, there will be no data loss, but be careful with splits. The
> > default split policy has been changed in this version if I am not wrong.
> It
> > causes some weird things.
> >
> > Thanks
> > Kiran
> >
> >
> > On Tue, Feb 26, 2013 at 1:03 PM, Yusup Ashrap <[EMAIL PROTECTED] (mailto:
> [EMAIL PROTECTED])> wrote:
> >
> > > hi all,
> > > I am updating production cluster from 0.90.2 to 0.94 . My table's size
> is
> > > about 20TB+ .
> > > Scheduled update includes upgrading both hbase,hadoop version,
> > > and I am also changing user with which I start up both hadoop and hbase
> > > cluster from user "A" to user "B".
> > >
> > > It's production environment , so I wanted know what steps I should not
> > > miss regarding this upgrade.
> > > Table is kinda big and I don't have backup cluster to backup my data.
> > > I wanted to know will there be a data loss scenario if I rollback after
> > > having failed to upgrade. thanks.
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Best Regards
> > > Yusup Ashrap
> > >
> >
> >
> >
> > --
> > Thank you
> > Kiran Sarvabhotla
> >
> > -----Even a correct decision is wrong when it is taken late
>
>

Hi Viral , thanks for reply.
Thing is that my table is kinda big. I have about one hour maximum downtime for this upgrade.
That is not enough to distcp 20TB data to another backup cluster, even I modify replication factor to 1,
and from my exp it's highly unlikely to succeed  for the first time with distcp.

thanks for the offer anyway.

--
Best Regards
Yusup Ashrap
On Tuesday, February 26, 2013 at 4:51 PM, Viral Bajaria wrote:

> Well if you can afford a longer downtime, you can always distcp your
> existing hbase data. This way if things get screwed up you can always
> restore a 0.90.x on that old backup. You cannot distcp while the cluster is
> running since it will not be able to get locks on file (I think I faced
> that issue but not sure since I did this upgrade mid-2012).
>
> When I did an upgrade, I brought up a test 0.94 cluster using the distcp
> backup (I took like 2-3 copies since I was very paranoid). Once I ran all
> my production jobs against the test instance and I was happy with all the
> results, that's when I re-ran the script to do the migration. It was all
> scripted out but I no longer have access to those scripts or else I would
> have shared it with you.
>
> Thanks,
> Viral
>
>

On Tue, Feb 26, 2013 at 12:00 AM, Yusup Ashrap <[EMAIL PROTECTED]> wrote:

> Hi Kiran , thanks for reply
> From what I've read from online docs , downtime is inevitable for
> upgrading from 0.90.2 to 0.94,
>

Yes.

Going from 0.90.x to 0.92., you will need to restart.

You will be able to do a rolling upgrade going from 0.92 to 0.94.

> and I can afford some downtime. I cannot afford dataloss, so I am
> concerning potential problems with rolling back to 0.90.2 if I fail to
> upgrade.
>
>
This is tough.  We do now support rolling back so as Viral suggests, it
would be ugly copy of data.   Search the mailing list.  There should be
description of doing background distcp and then doing a copy after the
shutdown to copy the differences only (try it first.. .I don't know of any
locks but if Viral says he had issue, I would pay attention).  Ask again if
no luck finding description...

St.Ack