Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Problem with recreation of a phantom table


+
Shumin Wu 2012-10-02, 22:55
+
yuzhihong@... 2012-10-02, 23:07
+
Shumin Wu 2012-10-02, 23:26
+
Kevin Odell 2012-10-03, 13:13
Copy link to this message
-
Re: Problem with recreation of a phantom table
Kevin,

Not at all preachy. Thank you very much for the all good, useful info. I
think your information will benefit others in the community if they would
meet the same problem I had. I wish I could see your post one day ago.

The cluster that I had that weird problem is a dev cluster and the issue
was blocking someone's work. I had to take the brutal force way to restore
it as quickly as possible. But yes you are right about in a production
deleting stuff without knowing the root cause is dangerous. So your
approach is very much desired in that case.

Cheers,

Shumin

On Wed, Oct 3, 2012 at 6:13 AM, Kevin O'dell <[EMAIL PROTECTED]>wrote:

> Shumin,
>
>   In the future for these kinds of issues please be more methodical about
> it.  It is important to collect information along the way, unless this
> hbase instance is something you don't care about losing.  I would
> recommend:
>
> hadoop fs -lsr /hbase > lsr_before.out
>
> hbase hbck -details 2>&1 | tee details_before.out
>
> Look at the details and try to understand what is happening, if you don't
> please submit your questions and concerns here while providing us data
>
> hadoop fs -cp /hbase/.META. /tmp/.META.
>
> hbase hbck -fixMeta -fixAssignments 2>&1 | tee fix.out
>
> Try to drop your table. If that doesn't work
>
> hadoop fs -rmr /hbase/phantom_table
>
> hbase hbck -fixMeta -fixAssignments 2>&1 | tee fix1.out
>
> This should correct the issue as it will rebuild META like the table was
> never there.  You may end up with some bad data cached for META so restart
> your master if you are having issues.  This way we can see what was done,
> follow along with data and be able to give complete diagnosis as well as
> file JIRAs to strengthen the product.
>
> Sorry for being preachy :)
>
> On Tue, Oct 2, 2012 at 7:26 PM, Shumin Wu <[EMAIL PROTECTED]> wrote:
>
> > Thanks for reply!
> >
> > I did a
> > hbase hbck -fixAssignments
> >
> > results showed
> > 0 inconsistencies detected.
> > Status: OK
> >
> > But the problem is still there -- hbase says table existing when I tried
> to
> > create and table not existed when i tried to delete.
> >
> > Then I brutally removed the phantom table from hdfs, rebooted the server,
> > and hoped to see the problem gone.
> >
> > But then met the NotAllMetaRegionsOnlineException error. So I did another
> > hbck.
> >
> > Then suddenly I was able to recreate the phantom table again. Problem
> > solved.
> >
> >
> > Shumin
> >
> > On Tue, Oct 2, 2012 at 4:07 PM, <[EMAIL PROTECTED]> wrote:
> >
> > > Can you try using hbck ?
> > >
> > > In the future, don't remove anything before using hbck.
> > >
> > > Thanks
> > >
> > >
> > >
> > > On Oct 2, 2012, at 3:55 PM, Shumin Wu <[EMAIL PROTECTED]> wrote:
> > >
> > > > Hi,
> > > >
> > > > I am using HBase 0.92 and got stuck with deletion/recreation of a
> > phantom
> > > > table. The table became "phantom" because hbase server went offline
> > > during
> > > > the first time when it got deleted. Since then I cannot recreate the
> > > table
> > > > because of the inconsistency catalog information.
> > > >
> > > > Below is what I got from the hbase shell (I replaced the table name
> > with
> > > a
> > > > fake name).
> > > >
> > > >
> > > > ============================================> > > > *
> > > > hbase(main):005:0> create 'phantom_table','cf'*
> > > >
> > > > ERROR: Table already exists: phantom_table!
> > > >
> > > > Here is some help for this command:
> > > > Create table; pass table name, a dictionary of specifications per
> > > > column family, and optionally a dictionary of table configuration.
> > > > Dictionaries are described below in the GENERAL NOTES section.
> > > > Examples:
> > > >
> > > >  hbase> create 't1', {NAME => 'f1', VERSIONS => 5}
> > > >  hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'}
> > > >  hbase> # The above in shorthand would be the following:
> > > >  hbase> create 't1', 'f1', 'f2', 'f3'
> > > >  hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000,