Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Problem with recreation of a phantom table


Copy link to this message
-
Re: Problem with recreation of a phantom table
Shumin,

  In the future for these kinds of issues please be more methodical about
it.  It is important to collect information along the way, unless this
hbase instance is something you don't care about losing.  I would recommend:

hadoop fs -lsr /hbase > lsr_before.out

hbase hbck -details 2>&1 | tee details_before.out

Look at the details and try to understand what is happening, if you don't
please submit your questions and concerns here while providing us data

hadoop fs -cp /hbase/.META. /tmp/.META.

hbase hbck -fixMeta -fixAssignments 2>&1 | tee fix.out

Try to drop your table. If that doesn't work

hadoop fs -rmr /hbase/phantom_table

hbase hbck -fixMeta -fixAssignments 2>&1 | tee fix1.out

This should correct the issue as it will rebuild META like the table was
never there.  You may end up with some bad data cached for META so restart
your master if you are having issues.  This way we can see what was done,
follow along with data and be able to give complete diagnosis as well as
file JIRAs to strengthen the product.

Sorry for being preachy :)

On Tue, Oct 2, 2012 at 7:26 PM, Shumin Wu <[EMAIL PROTECTED]> wrote:

> Thanks for reply!
>
> I did a
> hbase hbck -fixAssignments
>
> results showed
> 0 inconsistencies detected.
> Status: OK
>
> But the problem is still there -- hbase says table existing when I tried to
> create and table not existed when i tried to delete.
>
> Then I brutally removed the phantom table from hdfs, rebooted the server,
> and hoped to see the problem gone.
>
> But then met the NotAllMetaRegionsOnlineException error. So I did another
> hbck.
>
> Then suddenly I was able to recreate the phantom table again. Problem
> solved.
>
>
> Shumin
>
> On Tue, Oct 2, 2012 at 4:07 PM, <[EMAIL PROTECTED]> wrote:
>
> > Can you try using hbck ?
> >
> > In the future, don't remove anything before using hbck.
> >
> > Thanks
> >
> >
> >
> > On Oct 2, 2012, at 3:55 PM, Shumin Wu <[EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > I am using HBase 0.92 and got stuck with deletion/recreation of a
> phantom
> > > table. The table became "phantom" because hbase server went offline
> > during
> > > the first time when it got deleted. Since then I cannot recreate the
> > table
> > > because of the inconsistency catalog information.
> > >
> > > Below is what I got from the hbase shell (I replaced the table name
> with
> > a
> > > fake name).
> > >
> > >
> > > ============================================> > > *
> > > hbase(main):005:0> create 'phantom_table','cf'*
> > >
> > > ERROR: Table already exists: phantom_table!
> > >
> > > Here is some help for this command:
> > > Create table; pass table name, a dictionary of specifications per
> > > column family, and optionally a dictionary of table configuration.
> > > Dictionaries are described below in the GENERAL NOTES section.
> > > Examples:
> > >
> > >  hbase> create 't1', {NAME => 'f1', VERSIONS => 5}
> > >  hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'}
> > >  hbase> # The above in shorthand would be the following:
> > >  hbase> create 't1', 'f1', 'f2', 'f3'
> > >  hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000,
> > > BLOCKCACHE => true}
> > >  hbase> create 't1', 'f1', {SPLITS => ['10', '20', '30', '40']}
> > >  hbase> create 't1', 'f1', {SPLITS_FILE => 'splits.txt'}
> > >
> > > *
> > > hbase(main):006:0> disable 'phantom_table'*
> > >
> > > ERROR: Table phantom_table does not exist.'
> > >
> > > Here is some help for this command:
> > > Start disable of named table: e.g. "hbase> disable 't1'"
> > >
> > > ============================================> > >
> > > Here are the attempts I already did for investigation.
> > >
> > > - I did a fsck on /hbase. It reported healthy.
> > >
> > > - I did a scan on .META.. The phantom table is not listed there.
> > >
> > > - I checked zookeeper's hbase directory and found the phantom table. I
> > > deleted the entry but the problem reported above still persistent.
> > >
> > > Any comments or suggestions are highly appreciated!

Kevin O'Dell
Customer Operations Engineer, Cloudera
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB