Hi,

We recently encountered a few issues about hbase server incorrect shutdown causing table corruption issues. Basically, if the HBase server was not gracefully shutdown using stop-hbase.sh, instead it was shutdown by linux server reboot, we got '.META' table missing table rows. We were able to use 'hsbk -fixMeta -fixAssignment' to recover from this issue. However, I was wondering if this is the expected behavior of HBase that when hbase server is not gracefully shutdown, the database get into inconsistency state?  Is there any way from hbase configuration point of view to avoid this behavior?

Thanks,

YuLing

Which HBase version are you using ?

Did the HBase server experiencing shutdown carry the .META. table ?

Thanks

On Fri, Mar 15, 2013 at 3:45 PM, <[EMAIL PROTECTED]> wrote:

> Hi,
>
> We recently encountered a few issues about hbase server incorrect shutdown
> causing table corruption issues. Basically, if the HBase server was not
> gracefully shutdown using stop-hbase.sh, instead it was shutdown by linux
> server reboot, we got '.META' table missing table rows. We were able to use
> 'hsbk -fixMeta -fixAssignment' to recover from this issue. However, I was
> wondering if this is the expected behavior of HBase that when hbase server
> is not gracefully shutdown, the database get into inconsistency state?  Is
> there any way from hbase configuration point of view to avoid this behavior?
>
> Thanks,
>
> YuLing
>

I'm using 0.94.1.

We noticed that after HBase server shutdown, the .META. table lost a few rows of the tables. I had to use hbck to repair the .META. to get those rows back.  I was wondering if there could be any configuration that we didn't do properly to cause this issue...

Thanks,

YuLing

-----Original Message-----
From: Ted Yu [mailto:[EMAIL PROTECTED]]
Sent: Friday, March 15, 2013 3:51 PM
To: [EMAIL PROTECTED]
Subject: Re: hbase server incorrect shutdown case table corruption?

Which HBase version are you using ?

Did the HBase server experiencing shutdown carry the .META. table ?

Thanks

On Fri, Mar 15, 2013 at 3:45 PM, <[EMAIL PROTECTED]> wrote:

> Hi,
>
> We recently encountered a few issues about hbase server incorrect
> shutdown causing table corruption issues. Basically, if the HBase
> server was not gracefully shutdown using stop-hbase.sh, instead it was
> shutdown by linux server reboot, we got '.META' table missing table
> rows. We were able to use 'hsbk -fixMeta -fixAssignment' to recover
> from this issue. However, I was wondering if this is the expected
> behavior of HBase that when hbase server is not gracefully shutdown,
> the database get into inconsistency state?  Is there any way from hbase configuration point of view to avoid this behavior?
>
> Thanks,
>
> YuLing
>

Hi Ted,

We're using 0.94.1. Do you think if we upgrade to 0.94.5, the issue will be gone? I mean, HBase won't have this META inconsistence issue after incorrect shutdown?

We encountered quite a few major issues with 0.94.2, that's why we picked 0.94.1. Does that mean 0.94.5 is a pretty stable load?

Thanks,

YuLing

-----Original Message-----
From: Ted Yu [mailto:[EMAIL PROTECTED]]
Sent: Friday, March 15, 2013 3:51 PM
To: [EMAIL PROTECTED]
Subject: Re: hbase server incorrect shutdown case table corruption?

Which HBase version are you using ?

Did the HBase server experiencing shutdown carry the .META. table ?

Thanks

On Fri, Mar 15, 2013 at 3:45 PM, <[EMAIL PROTECTED]> wrote:

> Hi,
>
> We recently encountered a few issues about hbase server incorrect
> shutdown causing table corruption issues. Basically, if the HBase
> server was not gracefully shutdown using stop-hbase.sh, instead it was
> shutdown by linux server reboot, we got '.META' table missing table
> rows. We were able to use 'hsbk -fixMeta -fixAssignment' to recover
> from this issue. However, I was wondering if this is the expected
> behavior of HBase that when hbase server is not gracefully shutdown,
> the database get into inconsistency state?  Is there any way from hbase configuration point of view to avoid this behavior?
>
> Thanks,
>
> YuLing
>

Upgrading to 0.94.5 is definitely recommended.

As for inconsistencies in .META. Table, can you avoid improper shutdown ?

On Mar 15, 2013, at 5:36 PM, <[EMAIL PROTECTED]> wrote:

> Hi Ted,
>
> We're using 0.94.1. Do you think if we upgrade to 0.94.5, the issue will be gone? I mean, HBase won't have this META inconsistence issue after incorrect shutdown?
>
> We encountered quite a few major issues with 0.94.2, that's why we picked 0.94.1. Does that mean 0.94.5 is a pretty stable load?
>
> Thanks,
>
> YuLing
>
> -----Original Message-----
> From: Ted Yu [mailto:[EMAIL PROTECTED]]
> Sent: Friday, March 15, 2013 3:51 PM
> To: [EMAIL PROTECTED]
> Subject: Re: hbase server incorrect shutdown case table corruption?
>
> Which HBase version are you using ?
>
> Did the HBase server experiencing shutdown carry the .META. table ?
>
> Thanks
>
> On Fri, Mar 15, 2013 at 3:45 PM, <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> We recently encountered a few issues about hbase server incorrect
>> shutdown causing table corruption issues. Basically, if the HBase
>> server was not gracefully shutdown using stop-hbase.sh, instead it was
>> shutdown by linux server reboot, we got '.META' table missing table
>> rows. We were able to use 'hsbk -fixMeta -fixAssignment' to recover
>> from this issue. However, I was wondering if this is the expected
>> behavior of HBase that when hbase server is not gracefully shutdown,
>> the database get into inconsistency state?  Is there any way from hbase configuration point of view to avoid this behavior?
>>
>> Thanks,
>>
>> YuLing
>>

It's pretty hard for us to cover all the cases of improper shutdown... even if we try our best to cover all the cases, we can't avoid power outage case on customer sites...

Thanks,

YuLing

-----Original Message-----
From: Ted [mailto:[EMAIL PROTECTED]]
Sent: Friday, March 15, 2013 5:50 PM
To: [EMAIL PROTECTED]
Cc: <[EMAIL PROTECTED]>
Subject: Re: hbase server incorrect shutdown case table corruption?

Upgrading to 0.94.5 is definitely recommended.

As for inconsistencies in .META. Table, can you avoid improper shutdown ?

On Mar 15, 2013, at 5:36 PM, <[EMAIL PROTECTED]> wrote:

> Hi Ted,
>
> We're using 0.94.1. Do you think if we upgrade to 0.94.5, the issue will be gone? I mean, HBase won't have this META inconsistence issue after incorrect shutdown?
>
> We encountered quite a few major issues with 0.94.2, that's why we picked 0.94.1. Does that mean 0.94.5 is a pretty stable load?
>
> Thanks,
>
> YuLing
>
> -----Original Message-----
> From: Ted Yu [mailto:[EMAIL PROTECTED]]
> Sent: Friday, March 15, 2013 3:51 PM
> To: [EMAIL PROTECTED]
> Subject: Re: hbase server incorrect shutdown case table corruption?
>
> Which HBase version are you using ?
>
> Did the HBase server experiencing shutdown carry the .META. table ?
>
> Thanks
>
> On Fri, Mar 15, 2013 at 3:45 PM, <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> We recently encountered a few issues about hbase server incorrect
>> shutdown causing table corruption issues. Basically, if the HBase
>> server was not gracefully shutdown using stop-hbase.sh, instead it
>> was shutdown by linux server reboot, we got '.META' table missing
>> table rows. We were able to use 'hsbk -fixMeta -fixAssignment' to
>> recover from this issue. However, I was wondering if this is the
>> expected behavior of HBase that when hbase server is not gracefully
>> shutdown, the database get into inconsistency state?  Is there any way from hbase configuration point of view to avoid this behavior?
>>
>> Thanks,
>>
>> YuLing
>>

I think it is not acceptable to assume proper shutdown... if this is a real
issue we should fix it, unless it's already fixed on trunk.

On Fri, Mar 15, 2013 at 5:50 PM, Ted <[EMAIL PROTECTED]> wrote:

> Upgrading to 0.94.5 is definitely recommended.
>
> As for inconsistencies in .META. Table, can you avoid improper shutdown ?
>
> On Mar 15, 2013, at 5:36 PM, <[EMAIL PROTECTED]> wrote:
>
> > Hi Ted,
> >
> > We're using 0.94.1. Do you think if we upgrade to 0.94.5, the issue will
> be gone? I mean, HBase won't have this META inconsistence issue after
> incorrect shutdown?
> >
> > We encountered quite a few major issues with 0.94.2, that's why we
> picked 0.94.1. Does that mean 0.94.5 is a pretty stable load?
> >
> > Thanks,
> >
> > YuLing
> >
> > -----Original Message-----
> > From: Ted Yu [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, March 15, 2013 3:51 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: hbase server incorrect shutdown case table corruption?
> >
> > Which HBase version are you using ?
> >
> > Did the HBase server experiencing shutdown carry the .META. table ?
> >
> > Thanks
> >
> > On Fri, Mar 15, 2013 at 3:45 PM, <[EMAIL PROTECTED]> wrote:
> >
> >> Hi,
> >>
> >> We recently encountered a few issues about hbase server incorrect
> >> shutdown causing table corruption issues. Basically, if the HBase
> >> server was not gracefully shutdown using stop-hbase.sh, instead it was
> >> shutdown by linux server reboot, we got '.META' table missing table
> >> rows. We were able to use 'hsbk -fixMeta -fixAssignment' to recover
> >> from this issue. However, I was wondering if this is the expected
> >> behavior of HBase that when hbase server is not gracefully shutdown,
> >> the database get into inconsistency state?  Is there any way from hbase
> configuration point of view to avoid this behavior?
> >>
> >> Thanks,
> >>
> >> YuLing
> >>
>

So, I guess we should load the latest 0.94.5 and give it a try...if we still see the issue, we should report it as an HBase bug?

Thanks,

YuLing

-----Original Message-----
From: Sergey Shelukhin [mailto:[EMAIL PROTECTED]]
Sent: Friday, March 15, 2013 6:07 PM
To: [EMAIL PROTECTED]
Subject: Re: hbase server incorrect shutdown case table corruption?

I think it is not acceptable to assume proper shutdown... if this is a real issue we should fix it, unless it's already fixed on trunk.

On Fri, Mar 15, 2013 at 5:50 PM, Ted <[EMAIL PROTECTED]> wrote:

> Upgrading to 0.94.5 is definitely recommended.
>
> As for inconsistencies in .META. Table, can you avoid improper shutdown ?
>
> On Mar 15, 2013, at 5:36 PM, <[EMAIL PROTECTED]> wrote:
>
> > Hi Ted,
> >
> > We're using 0.94.1. Do you think if we upgrade to 0.94.5, the issue
> > will
> be gone? I mean, HBase won't have this META inconsistence issue after
> incorrect shutdown?
> >
> > We encountered quite a few major issues with 0.94.2, that's why we
> picked 0.94.1. Does that mean 0.94.5 is a pretty stable load?
> >
> > Thanks,
> >
> > YuLing
> >
> > -----Original Message-----
> > From: Ted Yu [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, March 15, 2013 3:51 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: hbase server incorrect shutdown case table corruption?
> >
> > Which HBase version are you using ?
> >
> > Did the HBase server experiencing shutdown carry the .META. table ?
> >
> > Thanks
> >
> > On Fri, Mar 15, 2013 at 3:45 PM, <[EMAIL PROTECTED]> wrote:
> >
> >> Hi,
> >>
> >> We recently encountered a few issues about hbase server incorrect
> >> shutdown causing table corruption issues. Basically, if the HBase
> >> server was not gracefully shutdown using stop-hbase.sh, instead it
> >> was shutdown by linux server reboot, we got '.META' table missing
> >> table rows. We were able to use 'hsbk -fixMeta -fixAssignment' to
> >> recover from this issue. However, I was wondering if this is the
> >> expected behavior of HBase that when hbase server is not gracefully
> >> shutdown, the database get into inconsistency state?  Is there any
> >> way from hbase
> configuration point of view to avoid this behavior?
> >>
> >> Thanks,
> >>
> >> YuLing
> >>
>

Sounds like a plan.

On Fri, Mar 15, 2013 at 6:22 PM, <[EMAIL PROTECTED]> wrote:

> So, I guess we should load the latest 0.94.5 and give it a try...if we
> still see the issue, we should report it as an HBase bug?
>
> Thanks,
>
> YuLing
>
> -----Original Message-----
> From: Sergey Shelukhin [mailto:[EMAIL PROTECTED]]
> Sent: Friday, March 15, 2013 6:07 PM
> To: [EMAIL PROTECTED]
> Subject: Re: hbase server incorrect shutdown case table corruption?
>
> I think it is not acceptable to assume proper shutdown... if this is a
> real issue we should fix it, unless it's already fixed on trunk.
>
> On Fri, Mar 15, 2013 at 5:50 PM, Ted <[EMAIL PROTECTED]> wrote:
>
> > Upgrading to 0.94.5 is definitely recommended.
> >
> > As for inconsistencies in .META. Table, can you avoid improper shutdown ?
> >
> > On Mar 15, 2013, at 5:36 PM, <[EMAIL PROTECTED]> wrote:
> >
> > > Hi Ted,
> > >
> > > We're using 0.94.1. Do you think if we upgrade to 0.94.5, the issue
> > > will
> > be gone? I mean, HBase won't have this META inconsistence issue after
> > incorrect shutdown?
> > >
> > > We encountered quite a few major issues with 0.94.2, that's why we
> > picked 0.94.1. Does that mean 0.94.5 is a pretty stable load?
> > >
> > > Thanks,
> > >
> > > YuLing
> > >
> > > -----Original Message-----
> > > From: Ted Yu [mailto:[EMAIL PROTECTED]]
> > > Sent: Friday, March 15, 2013 3:51 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: Re: hbase server incorrect shutdown case table corruption?
> > >
> > > Which HBase version are you using ?
> > >
> > > Did the HBase server experiencing shutdown carry the .META. table ?
> > >
> > > Thanks
> > >
> > > On Fri, Mar 15, 2013 at 3:45 PM, <[EMAIL PROTECTED]> wrote:
> > >
> > >> Hi,
> > >>
> > >> We recently encountered a few issues about hbase server incorrect
> > >> shutdown causing table corruption issues. Basically, if the HBase
> > >> server was not gracefully shutdown using stop-hbase.sh, instead it
> > >> was shutdown by linux server reboot, we got '.META' table missing
> > >> table rows. We were able to use 'hsbk -fixMeta -fixAssignment' to
> > >> recover from this issue. However, I was wondering if this is the
> > >> expected behavior of HBase that when hbase server is not gracefully
> > >> shutdown, the database get into inconsistency state?  Is there any
> > >> way from hbase
> > configuration point of view to avoid this behavior?
> > >>
> > >> Thanks,
> > >>
> > >> YuLing
> > >>
> >
>