I disable this table ivytest_deu , drop it .Error occurs.
ERROR: java.io.IOException: java.io.IOException: HTableDescriptor missing for ivytest_deu
at org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDescriptor(TableEventHandler.java:174)
at org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(DeleteTableHandler.java:44)
at org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)

Here is some help for this command:
Drop the named table. Table must first be disabled. If table has
more than one region, run a major compaction on .META.:

  hbase> major_compact ".META."

The major_compact ".META." doesn't work.
Then i try to create it ,but HBase says it .

ERROR: Table already exists: ivytest_deu!

After checking the region server log , the region server is always trying to load this region.

2012-10-16 00:00:00,308 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
2012-10-16 00:00:00,313 WARN org.apache.hadoop.hbase.util.FSTableDescriptors: The following folder is in HBase's root directory and doesn't contain a table descriptor, do consider deleting it: deu_ivytest
2012-10-16 00:00:00,358 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME => 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.', STARTKEY => '', ENDKEY => '', ENCODED => 985d6ca9986d7d8cfaf82daf523fcd45,}
2012-10-16 00:00:00,358 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol handler: region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45. protocol=com.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggregationProtocol
2012-10-16 00:00:00,358 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45., starting to roll back the global memstore size.
2012-10-16 00:00:00,358 INFO org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening of region {NAME => 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.', STARTKEY => '', ENDKEY => '', ENCODED => 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as FAILED_OPEN in ZK

And we have a endpoint in base .After the base tried to load this table ivy test_deu for 90,000 times ,the endpoint class also has been loaded for 90,000 times.
The jvm memory has been filled.
The gcutil shows
S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT  
34880.0 34880.0 34648.1  0.0   209472.0 209472.0 2792768.0  2792768.0  71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
34880.0 34880.0 34648.1  0.0   209472.0 209472.0 2792768.0  2792768.0  71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
34880.0 34880.0 34648.1  0.0   209472.0 209472.0 2792768.0  2792768.0  71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
34880.0 34880.0 34648.1  0.0   209472.0 209472.0 2792768.0  2792768.0  71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
34880.0 34880.0 34880.0  0.0   209472.0 209472.0 2792768.0  2792768.0  71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165
34880.0 34880.0 34880.0  0.0   209472.0 209472.0 2792768.0  2792768.0  71072.0 41461.5 129770 3448.191 24600 28481.974 31930.165

The jmap dump file shows

3982039 instances of class org.apache.hadoop.hbase.KeyValue
191050 instances of class org.apache.hadoop.fs.Path
187364 instances of class org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
187301 instances of class org.cliffc.high_scale_lib.Counter
102272 instances of class net.sf.ehcache.concurrent.ReadWriteLockSync
93652 instances of class org.apache.hadoop.hbase.HRegionInfo
93650 instances of class com.google.common.collect.MutableClassToInstanceMap
93650 instances of class DEUColumnAggregationEndpoint

DEUColumnAggregationEndpoint is our endpoint class.

We guess the 90,000 times check this table and load endpoint class leads this memory leak.

But how to drop this table?

Which version of HBase?
The below logs that you have attached says about a different table right '
deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
And the one you are trying to drop is ' ivytest_deu’

Regards
Ram

> -----Original Message-----
> From: 唐 颖 [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, October 16, 2012 1:23 PM
> To: [EMAIL PROTECTED]
> Subject: hbase can't drop a table
>
> I disable this table ivytest_deu , drop it .Error occurs.
>
>
> ERROR: java.io.IOException: java.io.IOException: HTableDescriptor
> missing for ivytest_deu
> at
> org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDescri
> ptor(TableEventHandler.java:174)
> at
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(Delete
> TableHandler.java:44)
> at
> org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> va:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> rImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEn
> gine.java:364)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:13
> 76)
>
> Here is some help for this command:
> Drop the named table. Table must first be disabled. If table has
> more than one region, run a major compaction on .META.:
>
>   hbase> major_compact ".META."
>
> The major_compact ".META." doesn't work.
> Then i try to create it ,but HBase says it .
>
> ERROR: Table already exists: ivytest_deu!
>
> After checking the region server log , the region server is always
> trying to load this region.
>
> 2012-10-16 00:00:00,308 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to
> open region:
> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> 2012-10-16 00:00:00,313 WARN
> org.apache.hadoop.hbase.util.FSTableDescriptors: The following folder
> is in HBase's root directory and doesn't contain a table descriptor, do
> consider deleting it: deu_ivytest
> 2012-10-16 00:00:00,358 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME =>
> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> STARTKEY => '', ENDKEY => '', ENCODED =>
> 985d6ca9986d7d8cfaf82daf523fcd45,}
> 2012-10-16 00:00:00,358 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol
> handler:
> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> protocol=com.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggreg
> ationProtocol
> 2012-10-16 00:00:00,358 ERROR
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
> open of
> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.,
> starting to roll back the global memstore size.
> 2012-10-16 00:00:00,358 INFO
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening
> of region {NAME =>
> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> STARTKEY => '', ENDKEY => '', ENCODED =>
> 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as FAILED_OPEN in ZK
>
> And we have a endpoint in base .After the base tried to load this table
> ivy test_deu for 90,000 times ,the endpoint class also has been loaded
> for 90,000 times.
> The jvm memory has been filled.
> The gcutil shows
> S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC
> PU    YGC     YGCT    FGC    FGCT     GCT
> 34880.0 34880.0 34648.1  0.0   209472.0 209472.0 2792768.0  2792768.0
> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> 34880.0 34880.0 34648.1  0.0   209472.0 209472.0 2792768.0  2792768.0
> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
> 34880.0 34880.0 34648.1  0.0   209472.0 209472.0 2792768.0  2792768.0
> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187

version 0.94.0, r8547

And the table is ivytest_deu.
在 2012-10-16，下午3:58，"Ramkrishna.S.Vasudevan" <[EMAIL PROTECTED]> 写道：

> Which version of HBase?
>
>
> The below logs that you have attached says about a different table right '
> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
> And the one you are trying to drop is ' ivytest_deu’
>
> Regards
> Ram
>
>
>
>> -----Original Message-----
>> From: 唐 颖 [mailto:[EMAIL PROTECTED]]
>> Sent: Tuesday, October 16, 2012 1:23 PM
>> To: [EMAIL PROTECTED]
>> Subject: hbase can't drop a table
>>
>> I disable this table ivytest_deu , drop it .Error occurs.
>>
>>
>> ERROR: java.io.IOException: java.io.IOException: HTableDescriptor
>> missing for ivytest_deu
>> at
>> org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDescri
>> ptor(TableEventHandler.java:174)
>> at
>> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(Delete
>> TableHandler.java:44)
>> at
>> org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
>> va:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
>> rImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEn
>> gine.java:364)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:13
>> 76)
>>
>> Here is some help for this command:
>> Drop the named table. Table must first be disabled. If table has
>> more than one region, run a major compaction on .META.:
>>
>>  hbase> major_compact ".META."
>>
>> The major_compact ".META." doesn't work.
>> Then i try to create it ,but HBase says it .
>>
>> ERROR: Table already exists: ivytest_deu!
>>
>> After checking the region server log , the region server is always
>> trying to load this region.
>>
>> 2012-10-16 00:00:00,308 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to
>> open region:
>> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
>> 2012-10-16 00:00:00,313 WARN
>> org.apache.hadoop.hbase.util.FSTableDescriptors: The following folder
>> is in HBase's root directory and doesn't contain a table descriptor, do
>> consider deleting it: deu_ivytest
>> 2012-10-16 00:00:00,358 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME =>
>> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
>> STARTKEY => '', ENDKEY => '', ENCODED =>
>> 985d6ca9986d7d8cfaf82daf523fcd45,}
>> 2012-10-16 00:00:00,358 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol
>> handler:
>> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
>> protocol=com.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggreg
>> ationProtocol
>> 2012-10-16 00:00:00,358 ERROR
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
>> open of
>> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.,
>> starting to roll back the global memstore size.
>> 2012-10-16 00:00:00,358 INFO
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening
>> of region {NAME =>
>> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
>> STARTKEY => '', ENDKEY => '', ENCODED =>
>> 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as FAILED_OPEN in ZK
>>
>> And we have a endpoint in base .After the base tried to load this table
>> ivy test_deu for 90,000 times ,the endpoint class also has been loaded
>> for 90,000 times.
>> The jvm memory has been filled.
>> The gcutil shows
>> S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC
>> PU    YGC     YGCT    FGC    FGCT     GCT
>> 34880.0 34880.0 34648.1  0.0   209472.0 209472.0 2792768.0  2792768.0
>> 71072.0 41461.5 129770 3448.191 24598 28469.996 31918.187
>> 34880.0 34880.0 34648.1  0.0   209472.0 209472.0 2792768.0  2792768.0

Hope this can help you!
https://issues.apache.org/jira/browse/HBASE-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418790#comment-13418790

Fowler Zhang

-----Original Message-----
From: 唐 颖 [mailto:[EMAIL PROTECTED]]
Sent: 2012年10月16日 16:08
To: [EMAIL PROTECTED]
Subject: Re: hbase can't drop a table

version 0.94.0, r8547

And the table is ivytest_deu.
在 2012-10-16，下午3:58，"Ramkrishna.S.Vasudevan" <[EMAIL PROTECTED]> 写道：

> Which version of HBase?
>
>
> The below logs that you have attached says about a different table right '
> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
> And the one you are trying to drop is ' ivytest_deu’
>
> Regards
> Ram
>
>
>
>> -----Original Message-----
>> From: 唐 颖 [mailto:[EMAIL PROTECTED]]
>> Sent: Tuesday, October 16, 2012 1:23 PM
>> To: [EMAIL PROTECTED]
>> Subject: hbase can't drop a table
>>
>> I disable this table ivytest_deu , drop it .Error occurs.
>>
>>
>> ERROR: java.io.IOException: java.io.IOException: HTableDescriptor
>> missing for ivytest_deu
>> at
>> org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDesc
>> ri
>> ptor(TableEventHandler.java:174)
>> at
>> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(Dele
>> te
>> TableHandler.java:44)
>> at
>> org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>> ja
>> va:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>> so
>> rImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpc
>> En
>> gine.java:364)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
>> 13
>> 76)
>>
>> Here is some help for this command:
>> Drop the named table. Table must first be disabled. If table has more
>> than one region, run a major compaction on .META.:
>>
>>  hbase> major_compact ".META."
>>
>> The major_compact ".META." doesn't work.
>> Then i try to create it ,but HBase says it .
>>
>> ERROR: Table already exists: ivytest_deu!
>>
>> After checking the region server log , the region server is always
>> trying to load this region.
>>
>> 2012-10-16 00:00:00,308 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Received request
>> to open region:
>> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
>> 2012-10-16 00:00:00,313 WARN
>> org.apache.hadoop.hbase.util.FSTableDescriptors: The following folder
>> is in HBase's root directory and doesn't contain a table descriptor,
>> do consider deleting it: deu_ivytest
>> 2012-10-16 00:00:00,358 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME
>> => 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
>> STARTKEY => '', ENDKEY => '', ENCODED =>
>> 985d6ca9986d7d8cfaf82daf523fcd45,}
>> 2012-10-16 00:00:00,358 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol
>> handler:
>> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
>> protocol=com.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggr
>> eg
>> ationProtocol
>> 2012-10-16 00:00:00,358 ERROR
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
>> Failed open of
>> region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.,
>> starting to roll back the global memstore size.
>> 2012-10-16 00:00:00,358 INFO
>> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
>> Opening of region {NAME =>
>> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
>> STARTKEY => '', ENDKEY => '', ENCODED =>
>> 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as FAILED_OPEN in
>> ZK
>>
>> And we have a endpoint in base .After the base tried to load this
>> table ivy test_deu for 90,000 times ,the endpoint class also has been

both ivytest_deu and deu_ivytest have the same problem.

On Tue, Oct 16, 2012 at 3:58 PM, Ramkrishna.S.Vasudevan <
[EMAIL PROTECTED]> wrote:

> Which version of HBase?
>
>
> The below logs that you have attached says about a different table right '
> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
> And the one you are trying to drop is ' ivytest_deu’
>
> Regards
> Ram
>
>
>
> > -----Original Message-----
> > From: 唐 颖 [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, October 16, 2012 1:23 PM
> > To: [EMAIL PROTECTED]
> > Subject: hbase can't drop a table
> >
> > I disable this table ivytest_deu , drop it .Error occurs.
> >
> >
> > ERROR: java.io.IOException: java.io.IOException: HTableDescriptor
> > missing for ivytest_deu
> >       at
> > org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDescri
> > ptor(TableEventHandler.java:174)
> >       at
> > org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(Delete
> > TableHandler.java:44)
> >       at
> > org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
> >       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >       at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> > va:39)
> >       at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> > rImpl.java:25)
> >       at java.lang.reflect.Method.invoke(Method.java:597)
> >       at
> > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEn
> > gine.java:364)
> >       at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:13
> > 76)
> >
> > Here is some help for this command:
> > Drop the named table. Table must first be disabled. If table has
> > more than one region, run a major compaction on .META.:
> >
> >   hbase> major_compact ".META."
> >
> > The major_compact ".META." doesn't work.
> > Then i try to create it ,but HBase says it .
> >
> > ERROR: Table already exists: ivytest_deu!
> >
> > After checking the region server log , the region server is always
> > trying to load this region.
> >
> > 2012-10-16 00:00:00,308 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to
> > open region:
> > deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> > 2012-10-16 00:00:00,313 WARN
> > org.apache.hadoop.hbase.util.FSTableDescriptors: The following folder
> > is in HBase's root directory and doesn't contain a table descriptor, do
> > consider deleting it: deu_ivytest
> > 2012-10-16 00:00:00,358 DEBUG
> > org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME =>
> > 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> > STARTKEY => '', ENDKEY => '', ENCODED =>
> > 985d6ca9986d7d8cfaf82daf523fcd45,}
> > 2012-10-16 00:00:00,358 DEBUG
> > org.apache.hadoop.hbase.regionserver.HRegion: Registered protocol
> > handler:
> > region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> > protocol=com.xingcloud.adhocprocessor.hbase.coprocessor.DEUColumnAggreg
> > ationProtocol
> > 2012-10-16 00:00:00,358 ERROR
> > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
> > open of
> > region=deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.,
> > starting to roll back the global memstore size.
> > 2012-10-16 00:00:00,358 INFO
> > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening
> > of region {NAME =>
> > 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> > STARTKEY => '', ENDKEY => '', ENCODED =>
> > 985d6ca9986d7d8cfaf82daf523fcd45,} failed, marking as FAILED_OPEN in ZK
> >
> > And we have a endpoint in base .After the base tried to load this table
> > ivy test_deu for 90,000 times ,the endpoint class also has been loaded
> > for 90,000 times.
> > The jvm memory has been filled.
> > The gcutil shows
> > S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC
> > PU    YGC     YGCT    FGC    FGCT     GCT
> > 34880.0 34880.0 34648.1  0.0   209472.0 209472.0 2792768.0  2792768.0
Best regards,

Ivy Tang

What does the 'list' command show?  Does it say the table exists or not?

What I can infer here is that the HTableDescriptor  file got deleted but the
META is having the entry.  Any chance of the HTD getting accidently deleted
in your cluster?

The hbck tool with -fixOrphanTables should atleast try to create the
HTableDescriptor file I suppose.  Then restart the cluster and then see what
happens.
I will not be able to access the logs even if you add it to pastebin.  But
pls do it so that some one else who has access can look into it.

Regards
Ram
> -----Original Message-----
> From: 张磊 [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, October 16, 2012 1:44 PM
> To: '[EMAIL PROTECTED]'
> Subject: RE: hbase can't drop a table
>
> Hope this can help you!
> https://issues.apache.org/jira/browse/HBASE-
> 3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=13418790#comment-13418790
>
> Fowler Zhang
>
> -----Original Message-----
> From: 唐 颖 [mailto:[EMAIL PROTECTED]]
> Sent: 2012年10月16日 16:08
> To: [EMAIL PROTECTED]
> Subject: Re: hbase can't drop a table
>
> version 0.94.0, r8547
>
> And the table is ivytest_deu.
>
>
> 在 2012-10-16，下午3:58，"Ramkrishna.S.Vasudevan"
> <[EMAIL PROTECTED]> 写道：
>
> > Which version of HBase?
> >
> >
> > The below logs that you have attached says about a different table
> right '
> > deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
> > And the one you are trying to drop is ' ivytest_deu’
> >
> > Regards
> > Ram
> >
> >
> >
> >> -----Original Message-----
> >> From: 唐 颖 [mailto:[EMAIL PROTECTED]]
> >> Sent: Tuesday, October 16, 2012 1:23 PM
> >> To: [EMAIL PROTECTED]
> >> Subject: hbase can't drop a table
> >>
> >> I disable this table ivytest_deu , drop it .Error occurs.
> >>
> >>
> >> ERROR: java.io.IOException: java.io.IOException: HTableDescriptor
> >> missing for ivytest_deu
> >> at
> >>
> org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDesc
> >> ri
> >> ptor(TableEventHandler.java:174)
> >> at
> >>
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.<init>(Dele
> >> te
> >> TableHandler.java:44)
> >> at
> >>
> org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1143)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >> ja
> >> va:39)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> >> so
> >> rImpl.java:25)
> >> at java.lang.reflect.Method.invoke(Method.java:597)
> >> at
> >>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpc
> >> En
> >> gine.java:364)
> >> at
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >> 13
> >> 76)
> >>
> >> Here is some help for this command:
> >> Drop the named table. Table must first be disabled. If table has
> more
> >> than one region, run a major compaction on .META.:
> >>
> >>  hbase> major_compact ".META."
> >>
> >> The major_compact ".META." doesn't work.
> >> Then i try to create it ,but HBase says it .
> >>
> >> ERROR: Table already exists: ivytest_deu!
> >>
> >> After checking the region server log , the region server is always
> >> trying to load this region.
> >>
> >> 2012-10-16 00:00:00,308 INFO
> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Received request
> >> to open region:
> >> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.
> >> 2012-10-16 00:00:00,313 WARN
> >> org.apache.hadoop.hbase.util.FSTableDescriptors: The following
> folder
> >> is in HBase's root directory and doesn't contain a table descriptor,
> >> do consider deleting it: deu_ivytest
> >> 2012-10-16 00:00:00,358 DEBUG
> >> org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME
> >> => 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> >> STARTKEY => '', ENDKEY => '', ENCODED =>
> >> 985d6ca9986d7d8cfaf82daf523fcd45,}

After checking the .META. table , the ivy test_deu and deu_ivytest entries do exist .

ROW                                                 COLUMN+CELL                                                                                                                                          
 ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c column=info:regioninfo, timestamp=1348821682041, value={NAME => 'ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c4c3e0277c.', STARTKEY => '', ENDKEY
 4c3e0277c.                                          => '', ENCODED => 77eb091b4753dd3b713f29c4c3e0277c,}                                                                                                
 ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c column=info:server, timestamp=1350032968219, value=ELEX-LA-WEB10:61020                                                                                
 4c3e0277c.                                                                                                                                                                                              
 ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c column=info:serverstartcode, timestamp=1350032968219, value=1350032549030                                                                            
 4c3e0277c.  

ROW                                                 COLUMN+CELL                                                                                                                                          
 deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da column=info:regioninfo, timestamp=1348826121970, value={NAME => 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.', STARTKEY => '', ENDKEY
 f523fcd45.                                          => '', ENCODED => 985d6ca9986d7d8cfaf82daf523fcd45,}                                                                                                
 deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da column=info:server, timestamp=1348826122164, value=ELEX-LA-WEB10:61020                                                                                
 f523fcd45.                                                                                                                                                                                              
 deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da column=info:serverstartcode, timestamp=1348826122164, value=1348648468768                                                                            
 f523fcd45.

And i deleted them from .META. table.Things seem Ok.

The region server won't try to load these regions.

Yes,it seems that the HTableDescriptor  file got deleted but the META is having the entry.
Thanks!
在 2012-10-16，下午4:52，"Ramkrishna.S.Vasudevan" <[EMAIL PROTECTED]> 写道：

> What does the 'list' command show?  Does it say the table exists or not?
>
> What I can infer here is that the HTableDescriptor  file got deleted but the
> META is having the entry.  Any chance of the HTD getting accidently deleted
> in your cluster?
>
> The hbck tool with -fixOrphanTables should atleast try to create the
> HTableDescriptor file I suppose.  Then restart the cluster and then see what
> happens.
> I will not be able to access the logs even if you add it to pastebin.  But
> pls do it so that some one else who has access can look into it.
>
> Regards
> Ram
>> -----Original Message-----
>> From: 张磊 [mailto:[EMAIL PROTECTED]]
>> Sent: Tuesday, October 16, 2012 1:44 PM
>> To: '[EMAIL PROTECTED]'
>> Subject: RE: hbase can't drop a table
>>
>> Hope this can help you!
>> https://issues.apache.org/jira/browse/HBASE-
>> 3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
>> tabpanel&focusedCommentId=13418790#comment-13418790
>>
>> Fowler Zhang
>>
>> -----Original Message-----
>> From: 唐 颖 [mailto:[EMAIL PROTECTED]]
>> Sent: 2012年10月16日 16:08
>> To: [EMAIL PROTECTED]
>> Subject: Re: hbase can't drop a table

If you get in that situation again:

1.) Verify that you don't have any remnants of the tables in HDFS
     hadoop fs -ls /hbase/

2.) If you do have any remnants and you don't care about these tables
     hadoop fs -mv /hbase/<table_name> /tmp

3.) ./bin/hbase hbck -fixMeta -fixAssignments

This should clean up META to match your HDFS and allow you to move forward.
 If not you may need a restart of HBase in case you had a RIT or something
like that hanging around.

On Tue, Oct 16, 2012 at 5:06 AM, 唐 颖 <[EMAIL PROTECTED]> wrote:

> After checking the .META. table , the ivy test_deu and deu_ivytest entries
> do exist .
>
> ROW                                                 COLUMN+CELL
>  ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c
> column=info:regioninfo, timestamp=1348821682041, value={NAME =>
> 'ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c4c3e0277c.', STARTKEY =>
> '', ENDKEY
>  4c3e0277c.                                          => '', ENCODED =>
> 77eb091b4753dd3b713f29c4c3e0277c,}
>  ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c column=info:server,
> timestamp=1350032968219, value=ELEX-LA-WEB10:61020
>  4c3e0277c.
>  ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c
> column=info:serverstartcode, timestamp=1350032968219, value=1350032549030
>  4c3e0277c.
>
> ROW                                                 COLUMN+CELL
>  deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da
> column=info:regioninfo, timestamp=1348826121970, value={NAME =>
> 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.', STARTKEY =>
> '', ENDKEY
>  f523fcd45.                                          => '', ENCODED =>
> 985d6ca9986d7d8cfaf82daf523fcd45,}
>  deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da column=info:server,
> timestamp=1348826122164, value=ELEX-LA-WEB10:61020
>  f523fcd45.
>  deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da
> column=info:serverstartcode, timestamp=1348826122164, value=1348648468768
>  f523fcd45.
>
> And i deleted them from .META. table.Things seem Ok.
>
> The region server won't try to load these regions.
>
> Yes,it seems that the HTableDescriptor  file got deleted but the META is
> having the entry.
>
>
> Thanks!
>
>
> 在 2012-10-16，下午4:52，"Ramkrishna.S.Vasudevan" <
> [EMAIL PROTECTED]> 写道：
>
> > What does the 'list' command show?  Does it say the table exists or not?
> >
> > What I can infer here is that the HTableDescriptor  file got deleted but
> the
> > META is having the entry.  Any chance of the HTD getting accidently
> deleted
> > in your cluster?
> >
> > The hbck tool with -fixOrphanTables should atleast try to create the
> > HTableDescriptor file I suppose.  Then restart the cluster and then see
> what
> > happens.
> > I will not be able to access the logs even if you add it to pastebin.
>  But
> > pls do it so that some one else who has access can look into it.
> >
> > Regards
> > Ram
> >> -----Original Message-----
> >> From: 张磊 [mailto:[EMAIL PROTECTED]]
> >> Sent: Tuesday, October 16, 2012 1:44 PM
> >> To: '[EMAIL PROTECTED]'
> >> Subject: RE: hbase can't drop a table
> >>
> >> Hope this can help you!
> >> https://issues.apache.org/jira/browse/HBASE-
> >> 3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> >> tabpanel&focusedCommentId=13418790#comment-13418790
> >>
> >> Fowler Zhang
> >>
> >> -----Original Message-----
> >> From: 唐 颖 [mailto:[EMAIL PROTECTED]]
> >> Sent: 2012年10月16日 16:08
> >> To: [EMAIL PROTECTED]
> >> Subject: Re: hbase can't drop a table
> >>
> >> version 0.94.0, r8547
> >>
> >> And the table is ivytest_deu.
> >>
> >>
> >> 在 2012-10-16，下午3:58，"Ramkrishna.S.Vasudevan"
> >> <[EMAIL PROTECTED]> 写道：
> >>
> >>> Which version of HBase?
> >>>
> >>>
> >>> The below logs that you have attached says about a different table
> >> right '
> >>> deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.'
> >>> And the one you are trying to drop is ' ivytest_deu’
> >>>
> >>> Regards
> >>> Ram
> >>>
> >

Kevin O'Dell
Customer Operations Engineer, Cloudera

Thanks for the write up Kevin.  Many users will be benefited by this.

Regards
Ram

> -----Original Message-----
> From: Kevin O'dell [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, October 16, 2012 9:42 PM
> To: [EMAIL PROTECTED]
> Subject: Re: hbase can't drop a table
>
> If you get in that situation again:
>
> 1.) Verify that you don't have any remnants of the tables in HDFS
>      hadoop fs -ls /hbase/
>
> 2.) If you do have any remnants and you don't care about these tables
>      hadoop fs -mv /hbase/<table_name> /tmp
>
> 3.) ./bin/hbase hbck -fixMeta -fixAssignments
>
> This should clean up META to match your HDFS and allow you to move
> forward.
>  If not you may need a restart of HBase in case you had a RIT or
> something
> like that hanging around.
>
> On Tue, Oct 16, 2012 at 5:06 AM, 唐 颖 <[EMAIL PROTECTED]> wrote:
>
> > After checking the .META. table , the ivy test_deu and deu_ivytest
> entries
> > do exist .
> >
> > ROW                                                 COLUMN+CELL
> >  ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c
> > column=info:regioninfo, timestamp=1348821682041, value={NAME =>
> > 'ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c4c3e0277c.',
> STARTKEY =>
> > '', ENDKEY
> >  4c3e0277c.                                          => '', ENCODED
> =>
> > 77eb091b4753dd3b713f29c4c3e0277c,}
> >  ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c
> column=info:server,
> > timestamp=1350032968219, value=ELEX-LA-WEB10:61020
> >  4c3e0277c.
> >  ivytest_deu,,1348821681817.77eb091b4753dd3b713f29c
> > column=info:serverstartcode, timestamp=1350032968219,
> value=1350032549030
> >  4c3e0277c.
> >
> > ROW                                                 COLUMN+CELL
> >  deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da
> > column=info:regioninfo, timestamp=1348826121970, value={NAME =>
> > 'deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82daf523fcd45.',
> STARTKEY =>
> > '', ENDKEY
> >  f523fcd45.                                          => '', ENCODED
> =>
> > 985d6ca9986d7d8cfaf82daf523fcd45,}
> >  deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da
> column=info:server,
> > timestamp=1348826122164, value=ELEX-LA-WEB10:61020
> >  f523fcd45.
> >  deu_ivytest,,1348826121781.985d6ca9986d7d8cfaf82da
> > column=info:serverstartcode, timestamp=1348826122164,
> value=1348648468768
> >  f523fcd45.
> >
> > And i deleted them from .META. table.Things seem Ok.
> >
> > The region server won't try to load these regions.
> >
> > Yes,it seems that the HTableDescriptor  file got deleted but the META
> is
> > having the entry.
> >
> >
> > Thanks!
> >
> >
> > 在 2012-10-16，下午4:52，"Ramkrishna.S.Vasudevan" <
> > [EMAIL PROTECTED]> 写道：
> >
> > > What does the 'list' command show?  Does it say the table exists or
> not?
> > >
> > > What I can infer here is that the HTableDescriptor  file got
> deleted but
> > the
> > > META is having the entry.  Any chance of the HTD getting accidently
> > deleted
> > > in your cluster?
> > >
> > > The hbck tool with -fixOrphanTables should atleast try to create
> the
> > > HTableDescriptor file I suppose.  Then restart the cluster and then
> see
> > what
> > > happens.
> > > I will not be able to access the logs even if you add it to
> pastebin.
> >  But
> > > pls do it so that some one else who has access can look into it.
> > >
> > > Regards
> > > Ram
> > >> -----Original Message-----
> > >> From: 张磊 [mailto:[EMAIL PROTECTED]]
> > >> Sent: Tuesday, October 16, 2012 1:44 PM
> > >> To: '[EMAIL PROTECTED]'
> > >> Subject: RE: hbase can't drop a table
> > >>
> > >> Hope this can help you!
> > >> https://issues.apache.org/jira/browse/HBASE-
> > >> 3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> > >> tabpanel&focusedCommentId=13418790#comment-13418790
> > >>
> > >> Fowler Zhang
> > >>
> > >> -----Original Message-----
> > >> From: 唐 颖 [mailto:[EMAIL PROTECTED]]
> > >> Sent: 2012年10月16日 16:08
> > >> To: [EMAIL PROTECTED]
> > >> Subject: Re: hbase can't drop a table