hi, hbase users.

I got a problem when I am testing coprocessors which is released at base 0.92.1.

here is the hbase shell outputs

----------------
hbase(main):001:0> truncate 'blog'
Truncating 'blog' table (it may take a while):

ERROR: Unknown table blog!

Here is some help for this command:
  Disables, drops and recreates the specified table.
hbase(main):002:0> list
TABLE                                                                                                                                                              
blog                                                                                                                                                              
counter                                                                                                                                                            
sidx                                                                                                                                                              
3 row(s) in 15.3610 seconds
----------------

yes, table 'blog' is a zombie I think.

there was some big stress from coprocessors which is using RegionServerObserver interfaces.

how could I fix this situation?

Hi Henry,

I faced the same issue not so long time ago...

Can you take a look at what you have in zookeeper under /hbase/table ?

If you table is there, that's why you see it on the list. Simply
remove it from zookeeper.

You can also take a look there: https://issues.apache.org/jira/browse/HBASE-6294

JM

2012/8/9, henry.kim <[EMAIL PROTECTED]>:
> hi, hbase users.
>
> I got a problem when I am testing coprocessors which is released at base
> 0.92.1.
>
> here is the hbase shell outputs
>
> ----------------
> hbase(main):001:0> truncate 'blog'
> Truncating 'blog' table (it may take a while):
>
> ERROR: Unknown table blog!
>
> Here is some help for this command:
>   Disables, drops and recreates the specified table.
>
>
> hbase(main):002:0> list
> TABLE
>
>
> blog
>
>
> counter
>
>
> sidx
>
>
> 3 row(s) in 15.3610 seconds
> ----------------
>
> yes, table 'blog' is a zombie I think.
>
> there was some big stress from coprocessors which is using
> RegionServerObserver interfaces.
>
> how could I fix this situation?

Hi Henry,

I would suggest you to try out 'hbck' utility of Hbase for dealing with
corrupt metadata in HBase. It's pretty effective and clean. If you use that
then you wont have to deal with manual cleanup most of the time.
Run: 'hbase hbck -help' command for more details.

Here is a useful link of HBase wiki:
http://hbase.apache.org/book/hbck.in.depth.html

HTH,
Anil Gupta

On Thu, Aug 9, 2012 at 5:00 AM, Jean-Marc Spaggiari <[EMAIL PROTECTED]
> wrote:

> Hi Henry,
>
> I faced the same issue not so long time ago...
>
> Can you take a look at what you have in zookeeper under /hbase/table ?
>
> If you table is there, that's why you see it on the list. Simply
> remove it from zookeeper.
>
> You can also take a look there:
> https://issues.apache.org/jira/browse/HBASE-6294
>
> JM
>
> 2012/8/9, henry.kim <[EMAIL PROTECTED]>:
> > hi, hbase users.
> >
> > I got a problem when I am testing coprocessors which is released at base
> > 0.92.1.
> >
> > here is the hbase shell outputs
> >
> > ----------------
> > hbase(main):001:0> truncate 'blog'
> > Truncating 'blog' table (it may take a while):
> >
> > ERROR: Unknown table blog!
> >
> > Here is some help for this command:
> >   Disables, drops and recreates the specified table.
> >
> >
> > hbase(main):002:0> list
> > TABLE
> >
> >
> > blog
> >
> >
> > counter
> >
> >
> > sidx
> >
> >
> > 3 row(s) in 15.3610 seconds
> > ----------------
> >
> > yes, table 'blog' is a zombie I think.
> >
> > there was some big stress from coprocessors which is using
> > RegionServerObserver interfaces.
> >
> > how could I fix this situation?
>

--
Thanks & Regards,
Anil Gupta

Hi, JM.

I followed procedures you mentioned.

here is it.

from zookeeper, I deleted it

---------------
$] rmr /hbase/table/*
$] ls /hbase/table
Node does not exist: /hbase/table
---------------

and then, I checked my hbase cluster.

---------------
$> hbase hbck
,,,,
Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.hbase.zookeeper.ZKTable.getDisabledOrDisablingTables(ZKTable.java:397)
at org.apache.hadoop.hbase.util.HBaseFsck$1.connect(HBaseFsck.java:996)
at org.apache.hadoop.hbase.util.HBaseFsck$1.connect(HBaseFsck.java:991)
at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:360)
at org.apache.hadoop.hbase.util.HBaseFsck.loadDisabledTables(HBaseFsck.java:991)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:360)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:383)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3121)
---------------

OMG.

ok, let's refresh master's cache which is from zk.

---------------
$> hadoop-hbase-master restart
$> hbase hbck
…
ERROR: Found inconsistency in table blog
Summary:
  -ROOT- is okay.
    Number of regions: 1
    Deployed on:  search-ddm-test10,60020,1344586183254
  .META. is okay.
    Number of regions: 1
    Deployed on:  search-ddm-test6,60020,1344586183295
Table blog is inconsistent.
    Number of regions: 3
    Deployed on:  search-ddm-test5,60020,1344586183269 search-ddm-test8,60020,1344586183257 search-ddm-test9,60020,1344586183260
8 inconsistencies detected.
Status: INCONSISTENT
---------------

now, it shows up the different situation for me. (thanks)

from hbase shell, I could check the existence of the tables I want to delete.

---------------
hbase(main):002:0> list
TABLE                                                                                                                                                              
blog                                                                                                                                                              
sidx                                                                                                                                                              
2 row(s) in 0.0180 seconds
---------------
try to fix it again.

---------------
$> hbase hbck -fix
…
Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
12/08/10 17:29:54 DEBUG util.HBaseFsck: There are 2 region info entries
Summary:
  -ROOT- is okay.
    Number of regions: 1
    Deployed on:  search-ddm-test10,60020,1344586183254
  .META. is okay.
    Number of regions: 1
    Deployed on:  search-ddm-test6,60020,1344586183295
0 inconsistencies detected.
Status: OK
---------------

hbase shell says also

---------------
hbase(main):009:0> list
TABLE                                                                                                                                                              
0 row(s) in 0.0160 seconds
---------------
not easy procedures to fix it  for me.
anyway, finally, in the conclusion, I fixed it.
2012. 8. 9., 오후 9:00, Jean-Marc Spaggiari <[EMAIL PROTECTED]> 작성:

> Hi Henry,
>
> I faced the same issue not so long time ago...
>
> Can you take a look at what you have in zookeeper under /hbase/table ?
>
> If you table is there, that's why you see it on the list. Simply
> remove it from zookeeper.
>
> You can also take a look there: https://issues.apache.org/jira/browse/HBASE-6294
>
> JM
>
> 2012/8/9, henry.kim <[EMAIL PROTECTED]>:
>> hi, hbase users.
>>
>> I got a problem when I am testing coprocessors which is released at base
>> 0.92.1.
>>
>> here is the hbase shell outputs
>>
>> ----------------
>> hbase(main):001:0> truncate 'blog'
>> Truncating 'blog' table (it may take a while):
>>
>> ERROR: Unknown table blog!
>>
>> Here is some help for this command:
>>  Disables, drops and recreates the specified table.

after the deletion did you restart HBase Master and region servers?

I Had the same problem a while ago and solve it by deleted from zk and
then restart.

Miguel Costa

On 08/10/2012 09:33 AM, Henry JunYoung KIM wrote:
> Hi, JM.
>
> I followed procedures you mentioned.
>
> here is it.
>
> from zookeeper, I deleted it
>
> ---------------
> $] rmr /hbase/table/*
> $] ls /hbase/table
> Node does not exist: /hbase/table
> ---------------
>
> and then, I checked my hbase cluster.
>
> ---------------
> $> hbase hbck
> ,,,,
> Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.zookeeper.ZKTable.getDisabledOrDisablingTables(ZKTable.java:397)
> at org.apache.hadoop.hbase.util.HBaseFsck$1.connect(HBaseFsck.java:996)
> at org.apache.hadoop.hbase.util.HBaseFsck$1.connect(HBaseFsck.java:991)
> at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:360)
> at org.apache.hadoop.hbase.util.HBaseFsck.loadDisabledTables(HBaseFsck.java:991)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:360)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:383)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3121)
> ---------------
>
> OMG.
>
> ok, let's refresh master's cache which is from zk.
>
> ---------------
> $> hadoop-hbase-master restart
> $> hbase hbck
> 占쏙옙
> ERROR: Found inconsistency in table blog
> Summary:
>   -ROOT- is okay.
>     Number of regions: 1
>     Deployed on:  search-ddm-test10,60020,1344586183254
>   .META. is okay.
>     Number of regions: 1
>     Deployed on:  search-ddm-test6,60020,1344586183295
> Table blog is inconsistent.
>     Number of regions: 3
>     Deployed on:  search-ddm-test5,60020,1344586183269 search-ddm-test8,60020,1344586183257 search-ddm-test9,60020,1344586183260
> 8 inconsistencies detected.
> Status: INCONSISTENT
> ---------------
>
> now, it shows up the different situation for me. (thanks)
>
> from hbase shell, I could check the existence of the tables I want to delete.
>
> ---------------
> hbase(main):002:0> list
> TABLE                                                                                                                                                              
> blog                                                                                                                                                              
> sidx                                                                                                                                                              
> 2 row(s) in 0.0180 seconds
> ---------------
> try to fix it again.
>
> ---------------
> $> hbase hbck -fix
> 占쏙옙
> Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
> 12/08/10 17:29:54 DEBUG util.HBaseFsck: There are 2 region info entries
> Summary:
>   -ROOT- is okay.
>     Number of regions: 1
>     Deployed on:  search-ddm-test10,60020,1344586183254
>   .META. is okay.
>     Number of regions: 1
>     Deployed on:  search-ddm-test6,60020,1344586183295
> 0 inconsistencies detected.
> Status: OK
> ---------------
>
> hbase shell says also
>
> ---------------
> hbase(main):009:0> list
> TABLE                                                                                                                                                              
> 0 row(s) in 0.0160 seconds
> ---------------
>
>
> not easy procedures to fix it  for me.
> anyway, finally, in the conclusion, I fixed it.
>
>
> 2012. 8. 9., 占쏙옙占쏙옙 9:00, Jean-Marc Spaggiari <[EMAIL PROTECTED]> 占쌜쇽옙:
>
>> Hi Henry,
>>
>> I faced the same issue not so long time ago...
>>
>> Can you take a look at what you have in zookeeper under /hbase/table ?
>>
>> If you table is there, that's why you see it on the list. Simply
>> remove it from zookeeper.
>>
>> You can also take a look there: https://issues.apache.org/jira/browse/HBASE-6294
>>

yes, it's a same situation for me.

----
iPhone에서 보냄

2012. 8. 10. 20:22 Miguel Costa <[EMAIL PROTECTED]> 작성:

> after the deletion did you restart HBase Master and region servers?
>
> I Had the same problem a while ago and solve it by deleted from zk and
> then restart.
>
> Miguel Costa
>
> On 08/10/2012 09:33 AM, Henry JunYoung KIM wrote:
>> Hi, JM.
>>
>> I followed procedures you mentioned.
>>
>> here is it.
>>
>> from zookeeper, I deleted it
>>
>> ---------------
>> $] rmr /hbase/table/*
>> $] ls /hbase/table
>> Node does not exist: /hbase/table
>> ---------------
>>
>> and then, I checked my hbase cluster.
>>
>> ---------------
>> $> hbase hbck
>> ,,,,
>> Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
>> Exception in thread "main" java.lang.NullPointerException
>>    at org.apache.hadoop.hbase.zookeeper.ZKTable.getDisabledOrDisablingTables(ZKTable.java:397)
>>    at org.apache.hadoop.hbase.util.HBaseFsck$1.connect(HBaseFsck.java:996)
>>    at org.apache.hadoop.hbase.util.HBaseFsck$1.connect(HBaseFsck.java:991)
>>    at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:360)
>>    at org.apache.hadoop.hbase.util.HBaseFsck.loadDisabledTables(HBaseFsck.java:991)
>>    at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:360)
>>    at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:383)
>>    at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3121)
>> ---------------
>>
>> OMG.
>>
>> ok, let's refresh master's cache which is from zk.
>>
>> ---------------
>> $> hadoop-hbase-master restart
>> $> hbase hbck
>> …
>> ERROR: Found inconsistency in table blog
>> Summary:
>>  -ROOT- is okay.
>>    Number of regions: 1
>>    Deployed on:  search-ddm-test10,60020,1344586183254
>>  .META. is okay.
>>    Number of regions: 1
>>    Deployed on:  search-ddm-test6,60020,1344586183295
>> Table blog is inconsistent.
>>    Number of regions: 3
>>    Deployed on:  search-ddm-test5,60020,1344586183269 search-ddm-test8,60020,1344586183257 search-ddm-test9,60020,1344586183260
>> 8 inconsistencies detected.
>> Status: INCONSISTENT
>> ---------------
>>
>> now, it shows up the different situation for me. (thanks)
>>
>> from hbase shell, I could check the existence of the tables I want to delete.
>>
>> ---------------
>> hbase(main):002:0> list
>> TABLE                                                                                                                                                              
>> blog                                                                                                                                                              
>> sidx                                                                                                                                                              
>> 2 row(s) in 0.0180 seconds
>> ---------------
>> try to fix it again.
>>
>> ---------------
>> $> hbase hbck -fix
>> …
>> Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
>> 12/08/10 17:29:54 DEBUG util.HBaseFsck: There are 2 region info entries
>> Summary:
>>  -ROOT- is okay.
>>    Number of regions: 1
>>    Deployed on:  search-ddm-test10,60020,1344586183254
>>  .META. is okay.
>>    Number of regions: 1
>>    Deployed on:  search-ddm-test6,60020,1344586183295
>> 0 inconsistencies detected.
>> Status: OK
>> ---------------
>>
>> hbase shell says also
>>
>> ---------------
>> hbase(main):009:0> list
>> TABLE                                                                                                                                                              
>> 0 row(s) in 0.0160 seconds
>> ---------------
>>
>>
>> not easy procedures to fix it  for me.
>> anyway, finally, in the conclusion, I fixed it.
>>
>>
>> 2012. 8. 9., 오후 9:00, Jean-Marc Spaggiari <[EMAIL PROTECTED]> 작성:
>>
>>> Hi Henry,
>>>
>>> I faced the same issue not so long time ago...
>>>
>>> Can you take a look at what you have in zookeeper under /hbase/table ?

You have deleted all the entries...

Only blog under table should have been deleted.

As proposed,  have you tried to restart everything, including zookeeper?
Le 10 août 2012 08:52, "GMAIL(henry)" <[EMAIL PROTECTED]> a écrit :

> yes, it's a same situation for me.
>
> ----
> iPhone에서 보냄
>
> 2012. 8. 10. 20:22 Miguel Costa <[EMAIL PROTECTED]> 작��:
>
> > after the deletion did you restart HBase Master and region servers?
> >
> > I Had the same problem a while ago and solve it by deleted from zk and
> > then restart.
> >
> > Miguel Costa
> >
> > On 08/10/2012 09:33 AM, Henry JunYoung KIM wrote:
> >> Hi, JM.
> >>
> >> I followed procedures you mentioned.
> >>
> >> here is it.
> >>
> >> from zookeeper, I deleted it
> >>
> >> ---------------
> >> $] rmr /hbase/table/*
> >> $] ls /hbase/table
> >> Node does not exist: /hbase/table
> >> ---------------
> >>
> >> and then, I checked my hbase cluster.
> >>
> >> ---------------
> >> $> hbase hbck
> >> ,,,,
> >> Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
> >> Exception in thread "main" java.lang.NullPointerException
> >>    at
> org.apache.hadoop.hbase.zookeeper.ZKTable.getDisabledOrDisablingTables(ZKTable.java:397)
> >>    at
> org.apache.hadoop.hbase.util.HBaseFsck$1.connect(HBaseFsck.java:996)
> >>    at
> org.apache.hadoop.hbase.util.HBaseFsck$1.connect(HBaseFsck.java:991)
> >>    at
> org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:360)
> >>    at
> org.apache.hadoop.hbase.util.HBaseFsck.loadDisabledTables(HBaseFsck.java:991)
> >>    at
> org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:360)
> >>    at
> org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:383)
> >>    at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3121)
> >> ---------------
> >>
> >> OMG.
> >>
> >> ok, let's refresh master's cache which is from zk.
> >>
> >> ---------------
> >> $> hadoop-hbase-master restart
> >> $> hbase hbck
> >> …
> >> ERROR: Found inconsistency in table blog
> >> Summary:
> >>  -ROOT- is okay.
> >>    Number of regions: 1
> >>    Deployed on:  search-ddm-test10,60020,1344586183254
> >>  .META. is okay.
> >>    Number of regions: 1
> >>    Deployed on:  search-ddm-test6,60020,1344586183295
> >> Table blog is inconsistent.
> >>    Number of regions: 3
> >>    Deployed on:  search-ddm-test5,60020,1344586183269
> search-ddm-test8,60020,1344586183257 search-ddm-test9,60020,1344586183260
> >> 8 inconsistencies detected.
> >> Status: INCONSISTENT
> >> ---------------
> >>
> >> now, it shows up the different situation for me. (thanks)
> >>
> >> from hbase shell, I could check the existence of the tables I want to
> delete.
> >>
> >> ---------------
> >> hbase(main):002:0> list
> >> TABLE
> >> blog
> >> sidx
> >> 2 row(s) in 0.0180 seconds
> >> ---------------
> >> try to fix it again.
> >>
> >> ---------------
> >> $> hbase hbck -fix
> >> …
> >> Number of empty REGIONINFO_QUALIFIER rows in .META.: 0
> >> 12/08/10 17:29:54 DEBUG util.HBaseFsck: There are 2 region info entries
> >> Summary:
> >>  -ROOT- is okay.
> >>    Number of regions: 1
> >>    Deployed on:  search-ddm-test10,60020,1344586183254
> >>  .META. is okay.
> >>    Number of regions: 1
> >>    Deployed on:  search-ddm-test6,60020,1344586183295
> >> 0 inconsistencies detected.
> >> Status: OK
> >> ---------------
> >>
> >> hbase shell says also
> >>
> >> ---------------
> >> hbase(main):009:0> list
> >> TABLE
> >> 0 row(s) in 0.0160 seconds
> >> ---------------
> >>
> >>
> >> not easy procedures to fix it  for me.
> >> anyway, finally, in the conclusion, I fixed it.
> >>
> >>
> >> 2012. 8. 9., 오후 9:00, Jean-Marc Spaggiari <[EMAIL PROTECTED]> 작성:
> >>
> >>> Hi Henry,
> >>>
> >>> I faced the same issue not so long time ago...
> >>>
> >>> Can you take a look at what you have in zookeeper under /hbase/table ?
> >>>
> >>> If you table is there, that's why you see it on the list. Simply
> >>> remove it from zookeeper.
> >>>
> >>> You can also take a look there: