Hi Devs,

I did an upgrade to HBase 0.92.2 recently. It ran fine for couple of days,
then I did online merge of regions. This got into a bit of inconsistent
state. To fix it the corruption I ran hbck and managed to get into
a situation described in
http://search-hadoop.com/m/sm2APK8FMo1&subj=Re+Wondering+what+hbck+should+do+in+this+situation.
After a bit of manual tinkering I got issues resolved.

But since then I see hbck reporting following errors (for different regions
at different times, not always the same region) consistently:

ERROR: Region { meta =>
table,c638103773063741088,1346207285319.d8051f85ab65b2e680ba5ebd383eaa08.,
hdfs => hdfs://
master-hadoop.example.net:8020/hbase/table/d8051f85ab65b2e680ba5ebd383eaa08<http://master-hadoop.ine-arp.rfiserve.net:8020/hbase/ine-arp/userprofile/d8051f85ab65b2e680ba5ebd383eaa08>,
deployed =>  } is a split parent in META, in HDFS, and not deployed on any
region server. This could be transient.

It does get back to 'Status: OK' state, but keeps flip flopping.

Any experiences?

-Shrijeet

Thanks for sharing your experience, Shrijeet.

Was the region cited below involved in your previous merging activity ?

Cheers

On Wed, Aug 29, 2012 at 9:36 AM, Shrijeet Paliwal
<[EMAIL PROTECTED]>wrote:

> Hi Devs,
>
> I did an upgrade to HBase 0.92.2 recently. It ran fine for couple of days,
> then I did online merge of regions. This got into a bit of inconsistent
> state. To fix it the corruption I ran hbck and managed to get into
> a situation described in
>
> http://search-hadoop.com/m/sm2APK8FMo1&subj=Re+Wondering+what+hbck+should+do+in+this+situation
> .
> After a bit of manual tinkering I got issues resolved.
>
> But since then I see hbck reporting following errors (for different regions
> at different times, not always the same region) consistently:
>
> ERROR: Region { meta =>
> table,c638103773063741088,1346207285319.d8051f85ab65b2e680ba5ebd383eaa08.,
> hdfs => hdfs://
> master-hadoop.example.net:8020/hbase/table/d8051f85ab65b2e680ba5ebd383eaa08
> <
> http://master-hadoop.ine-arp.rfiserve.net:8020/hbase/ine-arp/userprofile/d8051f85ab65b2e680ba5ebd383eaa08
> >,
> deployed =>  } is a split parent in META, in HDFS, and not deployed on any
> region server. This could be transient.
>
> It does get back to 'Status: OK' state, but keeps flip flopping.
>
> Any experiences?
>
> -Shrijeet
>

Ted,
Most likely Yes.

On Wed, Aug 29, 2012 at 9:41 AM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Thanks for sharing your experience, Shrijeet.
>
> Was the region cited below involved in your previous merging activity ?
>
> Cheers
>
> On Wed, Aug 29, 2012 at 9:36 AM, Shrijeet Paliwal
> <[EMAIL PROTECTED]>wrote:
>
> > Hi Devs,
> >
> > I did an upgrade to HBase 0.92.2 recently. It ran fine for couple of
> days,
> > then I did online merge of regions. This got into a bit of inconsistent
> > state. To fix it the corruption I ran hbck and managed to get into
> > a situation described in
> >
> >
> http://search-hadoop.com/m/sm2APK8FMo1&subj=Re+Wondering+what+hbck+should+do+in+this+situation
> > .
> > After a bit of manual tinkering I got issues resolved.
> >
> > But since then I see hbck reporting following errors (for different
> regions
> > at different times, not always the same region) consistently:
> >
> > ERROR: Region { meta =>
> >
> table,c638103773063741088,1346207285319.d8051f85ab65b2e680ba5ebd383eaa08.,
> > hdfs => hdfs://
> >
> master-hadoop.example.net:8020/hbase/table/d8051f85ab65b2e680ba5ebd383eaa08
> > <
> >
> http://master-hadoop.ine-arp.rfiserve.net:8020/hbase/ine-arp/userprofile/d8051f85ab65b2e680ba5ebd383eaa08
> > >,
> > deployed =>  } is a split parent in META, in HDFS, and not deployed on
> any
> > region server. This could be transient.
> >
> > It does get back to 'Status: OK' state, but keeps flip flopping.
> >
> > Any experiences?
> >
> > -Shrijeet
> >
>

Maybe you can dig up master / region server logs to see if there was any of
the following happening at time of merge ?
region server failover
load balancer run
zookeeper session expiration.

I feel we should make region merging more robust.

Cheers

On Wed, Aug 29, 2012 at 9:44 AM, Shrijeet Paliwal
<[EMAIL PROTECTED]>wrote:

> Ted,
> Most likely Yes.
>
> On Wed, Aug 29, 2012 at 9:41 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > Thanks for sharing your experience, Shrijeet.
> >
> > Was the region cited below involved in your previous merging activity ?
> >
> > Cheers
> >
> > On Wed, Aug 29, 2012 at 9:36 AM, Shrijeet Paliwal
> > <[EMAIL PROTECTED]>wrote:
> >
> > > Hi Devs,
> > >
> > > I did an upgrade to HBase 0.92.2 recently. It ran fine for couple of
> > days,
> > > then I did online merge of regions. This got into a bit of inconsistent
> > > state. To fix it the corruption I ran hbck and managed to get into
> > > a situation described in
> > >
> > >
> >
> http://search-hadoop.com/m/sm2APK8FMo1&subj=Re+Wondering+what+hbck+should+do+in+this+situation
> > > .
> > > After a bit of manual tinkering I got issues resolved.
> > >
> > > But since then I see hbck reporting following errors (for different
> > regions
> > > at different times, not always the same region) consistently:
> > >
> > > ERROR: Region { meta =>
> > >
> >
> table,c638103773063741088,1346207285319.d8051f85ab65b2e680ba5ebd383eaa08.,
> > > hdfs => hdfs://
> > >
> >
> master-hadoop.example.net:8020/hbase/table/d8051f85ab65b2e680ba5ebd383eaa08
> > > <
> > >
> >
> http://master-hadoop.ine-arp.rfiserve.net:8020/hbase/ine-arp/userprofile/d8051f85ab65b2e680ba5ebd383eaa08
> > > >,
> > > deployed =>  } is a split parent in META, in HDFS, and not deployed on
> > any
> > > region server. This could be transient.
> > >
> > > It does get back to 'Status: OK' state, but keeps flip flopping.
> > >
> > > Any experiences?
> > >
> > > -Shrijeet
> > >
> >
>