We have a small Hbase cluster on EC2 with 6 region servers. Lately we found
that the data in one of the column families is really not that useful for
us and decided to chuck it. This particular column family takes more than
50 percent of space on disk. We altered the table,removes the column family
and ran major compaction. We also ran major compaction on the '-ROOT-' and
the '.META.' tables. But there is still no reduction in total DFS file
size? Are we missing something here. Any help/pointers would be greatly
appreciated.

regards.

On Tue, Aug 7, 2012 at 7:19 AM, prem yadav <[EMAIL PROTECTED]> wrote:
> We have a small Hbase cluster on EC2 with 6 region servers. Lately we found
> that the data in one of the column families is really not that useful for
> us and decided to chuck it. This particular column family takes more than
> 50 percent of space on disk. We altered the table,removes the column family
> and ran major compaction. We also ran major compaction on the '-ROOT-' and
> the '.META.' tables. But there is still no reduction in total DFS file
> size? Are we missing something here. Any help/pointers would be greatly
> appreciated.
>

If you look in hdfs, you still see the column family?  (Look under
/hbase/TABLE_NAME/ENCODED_REGIONNAME...).  If you do an hdfs du -sh,
it occupies space still?

St.Ack

Hi Stack,
yes . I can still see the column family.

we ran the following command too:

fs -dus /hbase/<table name>/<region hash>/<cf that was deleted> and it
returned a large value which means that the space has not been reclaimed.
The hbase version we are running is 0.94.1.

is there  a manual way to force reclaim the space?

regards,
Prem

On Tue, Aug 7, 2012 at 12:52 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Tue, Aug 7, 2012 at 7:19 AM, prem yadav <[EMAIL PROTECTED]> wrote:
> > We have a small Hbase cluster on EC2 with 6 region servers. Lately we
> found
> > that the data in one of the column families is really not that useful for
> > us and decided to chuck it. This particular column family takes more than
> > 50 percent of space on disk. We altered the table,removes the column
> family
> > and ran major compaction. We also ran major compaction on the '-ROOT-'
> and
> > the '.META.' tables. But there is still no reduction in total DFS file
> > size? Are we missing something here. Any help/pointers would be greatly
> > appreciated.
> >
>
> If you look in hdfs, you still see the column family?  (Look under
> /hbase/TABLE_NAME/ENCODED_REGIONNAME...).  If you do an hdfs du -sh,
> it occupies space still?
>
> St.Ack
>

On Tue, Aug 7, 2012 at 9:59 AM, prem yadav <[EMAIL PROTECTED]> wrote:
> Hi Stack,
> yes . I can still see the column family.
>
> we ran the following command too:
>
> fs -dus /hbase/<table name>/<region hash>/<cf that was deleted> and it
> returned a large value which means that the space has not been reclaimed.
> The hbase version we are running is 0.94.1.
>
> is there  a manual way to force reclaim the space?
>

You could delete the column family from hdfs under each region.
Anything in the logs around the remove of the column family?  Maybe we
just don't do the delete cleanup in hdfs when we drop a cf (I've not
checked)?

St.Ack

Looking at TableDeleteFamilyHandler, I see:

  protected void handleTableOperation(List<HRegionInfo> hris) throws
IOException {
    // Update table descriptor in HDFS
    HTableDescriptor htd       this.masterServices.getMasterFileSystem().deleteColumn(tableName,
familyName);
    // Update in-memory descriptor cache
    this.masterServices.getTableDescriptors().add(htd);

MasterFileSystem.deleteColumn() calls:

    HTableDescriptor htd this.services.getTableDescriptors().get(tableName);
    htd.removeFamily(familyName);

I will dig some more.

But looks like there is a bug.

On Tue, Aug 7, 2012 at 9:21 AM, Stack <[EMAIL PROTECTED]> wrote:

> On Tue, Aug 7, 2012 at 9:59 AM, prem yadav <[EMAIL PROTECTED]> wrote:
> > Hi Stack,
> > yes . I can still see the column family.
> >
> > we ran the following command too:
> >
> > fs -dus /hbase/<table name>/<region hash>/<cf that was deleted> and it
> > returned a large value which means that the space has not been reclaimed.
> > The hbase version we are running is 0.94.1.
> >
> > is there  a manual way to force reclaim the space?
> >
>
> You could delete the column family from hdfs under each region.
> Anything in the logs around the remove of the column family?  Maybe we
> just don't do the delete cleanup in hdfs when we drop a cf (I've not
> checked)?
>
> St.Ack
>

Hi Stack,
we have got a lot of these in the logs after we deleted the column family.

org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
family ol does not exist in region <region>
        at
org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:4759)
        at
org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:2639)
        at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2065)
        at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1962)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3428)
        at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1386)
2012-08-05 00:11:37,697 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
rollbackMemstore rolled back 0 keyvalues from start:0 to end:1
On Tue, Aug 7, 2012 at 10:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Looking at TableDeleteFamilyHandler, I see:
>
>   protected void handleTableOperation(List<HRegionInfo> hris) throws
> IOException {
>     // Update table descriptor in HDFS
>     HTableDescriptor htd >       this.masterServices.getMasterFileSystem().deleteColumn(tableName,
> familyName);
>     // Update in-memory descriptor cache
>     this.masterServices.getTableDescriptors().add(htd);
>
> MasterFileSystem.deleteColumn() calls:
>
>     HTableDescriptor htd > this.services.getTableDescriptors().get(tableName);
>     htd.removeFamily(familyName);
>
> I will dig some more.
>
> But looks like there is a bug.
>
> On Tue, Aug 7, 2012 at 9:21 AM, Stack <[EMAIL PROTECTED]> wrote:
>
> > On Tue, Aug 7, 2012 at 9:59 AM, prem yadav <[EMAIL PROTECTED]> wrote:
> > > Hi Stack,
> > > yes . I can still see the column family.
> > >
> > > we ran the following command too:
> > >
> > > fs -dus /hbase/<table name>/<region hash>/<cf that was deleted> and it
> > > returned a large value which means that the space has not been
> reclaimed.
> > > The hbase version we are running is 0.94.1.
> > >
> > > is there  a manual way to force reclaim the space?
> > >
> >
> > You could delete the column family from hdfs under each region.
> > Anything in the logs around the remove of the column family?  Maybe we
> > just don't do the delete cleanup in hdfs when we drop a cf (I've not
> > checked)?
> >
> > St.Ack
> >
>

Hi

I see this atleast in trunk... in TableDeleteFamilyHandler's constructor..

 public TableDeleteFamilyHandler(byte[] tableName, byte [] familyName,

      Server server, final MasterServices masterServices) throwsIOException {

    super(EventType.C_M_ADD_FAMILY, tableName, server, masterServices);

    HTableDescriptor htd = getTableDescriptor();

    this.familyName = hasColumnFamily(htd, familyName);

  }
The event type is added as C_M_ADD_FAMILY....
May be this is the problem and it is there in 0.94.1 also !!!

./zahoor
On Wed, Aug 8, 2012 at 5:33 PM, prem yadav <[EMAIL PROTECTED]> wrote:

> Hi Stack,
> we have got a lot of these in the logs after we deleted the column family.
>
> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
> family ol does not exist in region <region>
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:4759)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:2639)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2065)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1962)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3428)
>         at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1386)
> 2012-08-05 00:11:37,697 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
> rollbackMemstore rolled back 0 keyvalues from start:0 to end:1
>
>
>
>
> On Tue, Aug 7, 2012 at 10:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > Looking at TableDeleteFamilyHandler, I see:
> >
> >   protected void handleTableOperation(List<HRegionInfo> hris) throws
> > IOException {
> >     // Update table descriptor in HDFS
> >     HTableDescriptor htd > >       this.masterServices.getMasterFileSystem().deleteColumn(tableName,
> > familyName);
> >     // Update in-memory descriptor cache
> >     this.masterServices.getTableDescriptors().add(htd);
> >
> > MasterFileSystem.deleteColumn() calls:
> >
> >     HTableDescriptor htd > > this.services.getTableDescriptors().get(tableName);
> >     htd.removeFamily(familyName);
> >
> > I will dig some more.
> >
> > But looks like there is a bug.
> >
> > On Tue, Aug 7, 2012 at 9:21 AM, Stack <[EMAIL PROTECTED]> wrote:
> >
> > > On Tue, Aug 7, 2012 at 9:59 AM, prem yadav <[EMAIL PROTECTED]>
> wrote:
> > > > Hi Stack,
> > > > yes . I can still see the column family.
> > > >
> > > > we ran the following command too:
> > > >
> > > > fs -dus /hbase/<table name>/<region hash>/<cf that was deleted> and
> it
> > > > returned a large value which means that the space has not been
> > reclaimed.
> > > > The hbase version we are running is 0.94.1.
> > > >
> > > > is there  a manual way to force reclaim the space?
> > > >
> > >
> > > You could delete the column family from hdfs under each region.
> > > Anything in the logs around the remove of the column family?  Maybe we
> > > just don't do the delete cleanup in hdfs when we drop a cf (I've not
> > > checked)?
> > >
> > > St.Ack
> > >
> >
>

Hi

Hmmm.. Finding more and more if i dig...
In file MasterFileSystem

 public HTableDescriptor deleteColumn(byte[] tableName, byte[] familyName)

      throws IOException {

    LOG.info("DeleteColumn. Table = " + Bytes.toString(tableName)

        + " family = " + Bytes.toString(familyName));

    HTableDescriptor htd = this.services
.getTableDescriptors().get(tableName);

    htd.removeFamily(familyName);

    this.services.getTableDescriptors().add(htd);

    return htd;

  }

it should have "this.services.getTableDescriptors().remove(htd)" instead of
"add".

I think this is the problem... the previous one will create a problem if
this was executed as a Executor service...

NOte: i still dont understand why DeleteColum was not submitted as a
executor service though.
./zahoor

On Thu, Aug 9, 2012 at 2:33 PM, J Mohamed Zahoor <[EMAIL PROTECTED]> wrote:

> Hi
>
> I see this atleast in trunk... in TableDeleteFamilyHandler's constructor..
>
>  public TableDeleteFamilyHandler(byte[] tableName, byte [] familyName,
>
>       Server server, final MasterServices masterServices) throwsIOException {
>
>     super(EventType.C_M_ADD_FAMILY, tableName, server, masterServices);
>
>     HTableDescriptor htd = getTableDescriptor();
>
>     this.familyName = hasColumnFamily(htd, familyName);
>
>   }
>
>
> The event type is added as C_M_ADD_FAMILY....
> May be this is the problem and it is there in 0.94.1 also !!!
>
> ./zahoor
>
>
> On Wed, Aug 8, 2012 at 5:33 PM, prem yadav <[EMAIL PROTECTED]> wrote:
>
>> Hi Stack,
>> we have got a lot of these in the logs after we deleted the column family.
>>
>> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
>> family ol does not exist in region <region>
>>         at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:4759)
>>         at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:2639)
>>         at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2065)
>>         at
>>
>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1962)
>>         at
>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3428)
>>         at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
>>         at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:601)
>>         at
>>
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>>         at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1386)
>> 2012-08-05 00:11:37,697 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion:
>> rollbackMemstore rolled back 0 keyvalues from start:0 to end:1
>>
>>
>>
>>
>> On Tue, Aug 7, 2012 at 10:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>
>> > Looking at TableDeleteFamilyHandler, I see:
>> >
>> >   protected void handleTableOperation(List<HRegionInfo> hris) throws
>> > IOException {
>> >     // Update table descriptor in HDFS
>> >     HTableDescriptor htd >> >       this.masterServices.getMasterFileSystem().deleteColumn(tableName,
>> > familyName);
>> >     // Update in-memory descriptor cache
>> >     this.masterServices.getTableDescriptors().add(htd);
>> >
>> > MasterFileSystem.deleteColumn() calls:
>> >
>> >     HTableDescriptor htd >> > this.services.getTableDescriptors().get(tableName);
>> >     htd.removeFamily(familyName);
>> >
>> > I will dig some more.
>> >
>> > But looks like there is a bug.
>> >
>> > On Tue, Aug 7, 2012 at 9:21 AM, Stack <[EMAIL PROTECTED]> wrote:
>> >
>> > > On Tue, Aug 7, 2012 at 9:59 AM, prem yadav <[EMAIL PROTECTED]>
>> wrote:
>> > > > Hi Stack,
>> > > > yes . I can still see the column family.
>> > > >
>> > > > we ran the following command too:
>> > > >
>> > > > fs -dus /hbase/<table name>/<region hash>/<cf that was deleted> and
>> it
>> > > > returned a large value which means that the space has not been

no.. The add call is necessary for Hmaster TableDescriptor cache update.

I see that the change is done only to the htable descriptors... a
FileSystem delete is missing. !!!
./zahoor
On Thu, Aug 9, 2012 at 3:05 PM, J Mohamed Zahoor <[EMAIL PROTECTED]> wrote:

> Hi
>
> Hmmm.. Finding more and more if i dig...
> In file MasterFileSystem
>
>  public HTableDescriptor deleteColumn(byte[] tableName, byte[] familyName)
>
>       throws IOException {
>
>     LOG.info("DeleteColumn. Table = " + Bytes.toString(tableName)
>
>         + " family = " + Bytes.toString(familyName));
>
>     HTableDescriptor htd = this.services
> .getTableDescriptors().get(tableName);
>
>     htd.removeFamily(familyName);
>
>     this.services.getTableDescriptors().add(htd);
>
>     return htd;
>
>   }
>
>
>
> it should have "this.services.getTableDescriptors().remove(htd)" instead
> of "add".
>
> I think this is the problem... the previous one will create a problem if
> this was executed as a Executor service...
>
>
>
> NOte: i still dont understand why DeleteColum was not submitted as a
> executor service though.
>
>
> ./zahoor
>
>
>
> On Thu, Aug 9, 2012 at 2:33 PM, J Mohamed Zahoor <[EMAIL PROTECTED]> wrote:
>
>> Hi
>>
>> I see this atleast in trunk... in TableDeleteFamilyHandler's constructor..
>>
>>  public TableDeleteFamilyHandler(byte[] tableName, byte [] familyName,
>>
>>       Server server, final MasterServices masterServices) throwsIOException {
>>
>>     super(EventType.C_M_ADD_FAMILY, tableName, server, masterServices);
>>
>>     HTableDescriptor htd = getTableDescriptor();
>>
>>     this.familyName = hasColumnFamily(htd, familyName);
>>
>>   }
>>
>>
>> The event type is added as C_M_ADD_FAMILY....
>> May be this is the problem and it is there in 0.94.1 also !!!
>>
>> ./zahoor
>>
>>
>> On Wed, Aug 8, 2012 at 5:33 PM, prem yadav <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Stack,
>>> we have got a lot of these in the logs after we deleted the column
>>> family.
>>>
>>> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
>>> family ol does not exist in region <region>
>>>         at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:4759)
>>>         at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:2639)
>>>         at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2065)
>>>         at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1962)
>>>         at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3428)
>>>         at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
>>>         at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>         at java.lang.reflect.Method.invoke(Method.java:601)
>>>         at
>>>
>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>>>         at
>>>
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1386)
>>> 2012-08-05 00:11:37,697 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>> rollbackMemstore rolled back 0 keyvalues from start:0 to end:1
>>>
>>>
>>>
>>>
>>> On Tue, Aug 7, 2012 at 10:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>>
>>> > Looking at TableDeleteFamilyHandler, I see:
>>> >
>>> >   protected void handleTableOperation(List<HRegionInfo> hris) throws
>>> > IOException {
>>> >     // Update table descriptor in HDFS
>>> >     HTableDescriptor htd >>> >       this.masterServices.getMasterFileSystem().deleteColumn(tableName,
>>> > familyName);
>>> >     // Update in-memory descriptor cache
>>> >     this.masterServices.getTableDescriptors().add(htd);
>>> >
>>> > MasterFileSystem.deleteColumn() calls:
>>> >
>>> >     HTableDescriptor htd >>> > this.services.getTableDescriptors().get(tableName);
>>> >     htd.removeFamily(familyName);
>>> >
>>> > I will dig some more.
>>> >
>>> > But looks like there is a bug.

Hi zahoor,

could you file a jira with what you found? It looks like a bug.
Thanks.
-- Lars

----- Original Message -----
From: J Mohamed Zahoor <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Cc:
Sent: Thursday, August 9, 2012 2:03 AM
Subject: Re: Hbase- Hadoop DFS size not decreased even after deleting a column family

Hi

I see this atleast in trunk... in TableDeleteFamilyHandler's constructor..

public TableDeleteFamilyHandler(byte[] tableName, byte [] familyName,

      Server server, final MasterServices masterServices) throwsIOException {

    super(EventType.C_M_ADD_FAMILY, tableName, server, masterServices);

    HTableDescriptor htd = getTableDescriptor();

    this.familyName = hasColumnFamily(htd, familyName);

  }
The event type is added as C_M_ADD_FAMILY....
May be this is the problem and it is there in 0.94.1 also !!!

./zahoor
On Wed, Aug 8, 2012 at 5:33 PM, prem yadav <[EMAIL PROTECTED]> wrote:

> Hi Stack,
> we have got a lot of these in the logs after we deleted the column family.
>
> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
> family ol does not exist in region <region>
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:4759)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:2639)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2065)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1962)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3428)
>         at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1386)
> 2012-08-05 00:11:37,697 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
> rollbackMemstore rolled back 0 keyvalues from start:0 to end:1
>
>
>
>
> On Tue, Aug 7, 2012 at 10:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > Looking at TableDeleteFamilyHandler, I see:
> >
> >   protected void handleTableOperation(List<HRegionInfo> hris) throws
> > IOException {
> >     // Update table descriptor in HDFS
> >     HTableDescriptor htd > >       this.masterServices.getMasterFileSystem().deleteColumn(tableName,
> > familyName);
> >     // Update in-memory descriptor cache
> >     this.masterServices.getTableDescriptors().add(htd);
> >
> > MasterFileSystem.deleteColumn() calls:
> >
> >     HTableDescriptor htd > > this.services.getTableDescriptors().get(tableName);
> >     htd.removeFamily(familyName);
> >
> > I will dig some more.
> >
> > But looks like there is a bug.
> >
> > On Tue, Aug 7, 2012 at 9:21 AM, Stack <[EMAIL PROTECTED]> wrote:
> >
> > > On Tue, Aug 7, 2012 at 9:59 AM, prem yadav <[EMAIL PROTECTED]>
> wrote:
> > > > Hi Stack,
> > > > yes . I can still see the column family.
> > > >
> > > > we ran the following command too:
> > > >
> > > > fs -dus /hbase/<table name>/<region hash>/<cf that was deleted> and
> it
> > > > returned a large value which means that the space has not been
> > reclaimed.
> > > > The hbase version we are running is 0.94.1.
> > > >
> > > > is there  a manual way to force reclaim the space?
> > > >
> > >
> > > You could delete the column family from hdfs under each region.
> > > Anything in the logs around the remove of the column family?  Maybe we
> > > just don't do the delete cleanup in hdfs when we drop a cf (I've not
> > > checked)?
> > >
> > > St.Ack
> > >
> >
>

Hi Lars,

Will file it...

./Zahoor

On Fri, Aug 10, 2012 at 12:00 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> Hi zahoor,
>
> could you file a jira with what you found? It looks like a bug.
> Thanks.
>
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: J Mohamed Zahoor <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc:
> Sent: Thursday, August 9, 2012 2:03 AM
> Subject: Re: Hbase- Hadoop DFS size not decreased even after deleting a
> column family
>
> Hi
>
> I see this atleast in trunk... in TableDeleteFamilyHandler's constructor..
>
> public TableDeleteFamilyHandler(byte[] tableName, byte [] familyName,
>
>       Server server, final MasterServices masterServices)
> throwsIOException {
>
>     super(EventType.C_M_ADD_FAMILY, tableName, server, masterServices);
>
>     HTableDescriptor htd = getTableDescriptor();
>
>     this.familyName = hasColumnFamily(htd, familyName);
>
>   }
>
>
> The event type is added as C_M_ADD_FAMILY....
> May be this is the problem and it is there in 0.94.1 also !!!
>
> ./zahoor
>
>
> On Wed, Aug 8, 2012 at 5:33 PM, prem yadav <[EMAIL PROTECTED]> wrote:
>
> > Hi Stack,
> > we have got a lot of these in the logs after we deleted the column
> family.
> >
> > org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
> > family ol does not exist in region <region>
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:4759)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:2639)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2065)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1962)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3428)
> >         at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> >         at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:601)
> >         at
> >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> >         at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1386)
> > 2012-08-05 00:11:37,697 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion:
> > rollbackMemstore rolled back 0 keyvalues from start:0 to end:1
> >
> >
> >
> >
> > On Tue, Aug 7, 2012 at 10:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> > > Looking at TableDeleteFamilyHandler, I see:
> > >
> > >   protected void handleTableOperation(List<HRegionInfo> hris) throws
> > > IOException {
> > >     // Update table descriptor in HDFS
> > >     HTableDescriptor htd > > >       this.masterServices.getMasterFileSystem().deleteColumn(tableName,
> > > familyName);
> > >     // Update in-memory descriptor cache
> > >     this.masterServices.getTableDescriptors().add(htd);
> > >
> > > MasterFileSystem.deleteColumn() calls:
> > >
> > >     HTableDescriptor htd > > > this.services.getTableDescriptors().get(tableName);
> > >     htd.removeFamily(familyName);
> > >
> > > I will dig some more.
> > >
> > > But looks like there is a bug.
> > >
> > > On Tue, Aug 7, 2012 at 9:21 AM, Stack <[EMAIL PROTECTED]> wrote:
> > >
> > > > On Tue, Aug 7, 2012 at 9:59 AM, prem yadav <[EMAIL PROTECTED]>
> > wrote:
> > > > > Hi Stack,
> > > > > yes . I can still see the column family.
> > > > >
> > > > > we ran the following command too:
> > > > >
> > > > > fs -dus /hbase/<table name>/<region hash>/<cf that was deleted> and
> > it
> > > > > returned a large value which means that the space has not been
> > > reclaimed.
> > > > > The hbase version we are running is 0.94.1.
> > > > >
> > > > > is there  a manual way to force reclaim the space?
> > > > >
> > > >
> > > > You could delete the column family from hdfs under each region.
> > > > Anything in the logs around the remove of the column family?  Maybe

Look at this post for more about Catalog Janitor

http://blog.zahoor.in/2012/08/hbase-hmaster-architecture/

./zahoor

On Fri, Aug 10, 2012 at 12:47 PM, J Mohamed Zahoor <[EMAIL PROTECTED]> wrote:

> Hi Lars,
>
> Will file it...
>
> ./Zahoor
>
>
> On Fri, Aug 10, 2012 at 12:00 AM, lars hofhansl <[EMAIL PROTECTED]>wrote:
>
>> Hi zahoor,
>>
>> could you file a jira with what you found? It looks like a bug.
>> Thanks.
>>
>>
>> -- Lars
>>
>>
>>
>> ----- Original Message -----
>> From: J Mohamed Zahoor <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Cc:
>> Sent: Thursday, August 9, 2012 2:03 AM
>> Subject: Re: Hbase- Hadoop DFS size not decreased even after deleting a
>> column family
>>
>> Hi
>>
>> I see this atleast in trunk... in TableDeleteFamilyHandler's constructor..
>>
>> public TableDeleteFamilyHandler(byte[] tableName, byte [] familyName,
>>
>>       Server server, final MasterServices masterServices)
>> throwsIOException {
>>
>>     super(EventType.C_M_ADD_FAMILY, tableName, server, masterServices);
>>
>>     HTableDescriptor htd = getTableDescriptor();
>>
>>     this.familyName = hasColumnFamily(htd, familyName);
>>
>>   }
>>
>>
>> The event type is added as C_M_ADD_FAMILY....
>> May be this is the problem and it is there in 0.94.1 also !!!
>>
>> ./zahoor
>>
>>
>> On Wed, Aug 8, 2012 at 5:33 PM, prem yadav <[EMAIL PROTECTED]> wrote:
>>
>> > Hi Stack,
>> > we have got a lot of these in the logs after we deleted the column
>> family.
>> >
>> > org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
>> > family ol does not exist in region <region>
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:4759)
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:2639)
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2065)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1962)
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3428)
>> >         at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
>> >         at
>> >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >         at java.lang.reflect.Method.invoke(Method.java:601)
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>> >         at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1386)
>> > 2012-08-05 00:11:37,697 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion:
>> > rollbackMemstore rolled back 0 keyvalues from start:0 to end:1
>> >
>> >
>> >
>> >
>> > On Tue, Aug 7, 2012 at 10:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> >
>> > > Looking at TableDeleteFamilyHandler, I see:
>> > >
>> > >   protected void handleTableOperation(List<HRegionInfo> hris) throws
>> > > IOException {
>> > >     // Update table descriptor in HDFS
>> > >     HTableDescriptor htd >> > >
>>  this.masterServices.getMasterFileSystem().deleteColumn(tableName,
>> > > familyName);
>> > >     // Update in-memory descriptor cache
>> > >     this.masterServices.getTableDescriptors().add(htd);
>> > >
>> > > MasterFileSystem.deleteColumn() calls:
>> > >
>> > >     HTableDescriptor htd >> > > this.services.getTableDescriptors().get(tableName);
>> > >     htd.removeFamily(familyName);
>> > >
>> > > I will dig some more.
>> > >
>> > > But looks like there is a bug.
>> > >
>> > > On Tue, Aug 7, 2012 at 9:21 AM, Stack <[EMAIL PROTECTED]> wrote:
>> > >
>> > > > On Tue, Aug 7, 2012 at 9:59 AM, prem yadav <[EMAIL PROTECTED]>
>> > wrote:
>> > > > > Hi Stack,
>> > > > > yes . I can still see the column family.
>> > > > >
>> > > > > we ran the following command too:
>> > > > >
>> > > > > fs -dus /hbase/<table name>/<region hash>/<cf that was deleted>
>> and
>> > it
>> > > > > returned a large value which means that the space has not been

Sorry.. previous mail Not for this thread.

./zahoor

On Fri, Aug 10, 2012 at 12:54 PM, J Mohamed Zahoor <[EMAIL PROTECTED]> wrote:

> Look at this post for more about Catalog Janitor
>
> http://blog.zahoor.in/2012/08/hbase-hmaster-architecture/
>
> ./zahoor
>
>
> On Fri, Aug 10, 2012 at 12:47 PM, J Mohamed Zahoor <[EMAIL PROTECTED]>wrote:
>
>> Hi Lars,
>>
>> Will file it...
>>
>> ./Zahoor
>>
>>
>> On Fri, Aug 10, 2012 at 12:00 AM, lars hofhansl <[EMAIL PROTECTED]>wrote:
>>
>>> Hi zahoor,
>>>
>>> could you file a jira with what you found? It looks like a bug.
>>> Thanks.
>>>
>>>
>>> -- Lars
>>>
>>>
>>>
>>> ----- Original Message -----
>>> From: J Mohamed Zahoor <[EMAIL PROTECTED]>
>>> To: [EMAIL PROTECTED]
>>> Cc:
>>> Sent: Thursday, August 9, 2012 2:03 AM
>>> Subject: Re: Hbase- Hadoop DFS size not decreased even after deleting a
>>> column family
>>>
>>> Hi
>>>
>>> I see this atleast in trunk... in TableDeleteFamilyHandler's
>>> constructor..
>>>
>>> public TableDeleteFamilyHandler(byte[] tableName, byte [] familyName,
>>>
>>>       Server server, final MasterServices masterServices)
>>> throwsIOException {
>>>
>>>     super(EventType.C_M_ADD_FAMILY, tableName, server, masterServices);
>>>
>>>     HTableDescriptor htd = getTableDescriptor();
>>>
>>>     this.familyName = hasColumnFamily(htd, familyName);
>>>
>>>   }
>>>
>>>
>>> The event type is added as C_M_ADD_FAMILY....
>>> May be this is the problem and it is there in 0.94.1 also !!!
>>>
>>> ./zahoor
>>>
>>>
>>> On Wed, Aug 8, 2012 at 5:33 PM, prem yadav <[EMAIL PROTECTED]> wrote:
>>>
>>> > Hi Stack,
>>> > we have got a lot of these in the logs after we deleted the column
>>> family.
>>> >
>>> > org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException:
>>> Column
>>> > family ol does not exist in region <region>
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:4759)
>>> >         at
>>> >
>>> >
>>> org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:2639)
>>> >         at
>>> >
>>> >
>>> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2065)
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1962)
>>> >         at
>>> >
>>> >
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3428)
>>> >         at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
>>> >         at
>>> >
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> >         at java.lang.reflect.Method.invoke(Method.java:601)
>>> >         at
>>> >
>>> >
>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1386)
>>> > 2012-08-05 00:11:37,697 DEBUG
>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>> > rollbackMemstore rolled back 0 keyvalues from start:0 to end:1
>>> >
>>> >
>>> >
>>> >
>>> > On Tue, Aug 7, 2012 at 10:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>> >
>>> > > Looking at TableDeleteFamilyHandler, I see:
>>> > >
>>> > >   protected void handleTableOperation(List<HRegionInfo> hris) throws
>>> > > IOException {
>>> > >     // Update table descriptor in HDFS
>>> > >     HTableDescriptor htd >>> > >
>>>  this.masterServices.getMasterFileSystem().deleteColumn(tableName,
>>> > > familyName);
>>> > >     // Update in-memory descriptor cache
>>> > >     this.masterServices.getTableDescriptors().add(htd);
>>> > >
>>> > > MasterFileSystem.deleteColumn() calls:
>>> > >
>>> > >     HTableDescriptor htd >>> > > this.services.getTableDescriptors().get(tableName);
>>> > >     htd.removeFamily(familyName);
>>> > >
>>> > > I will dig some more.
>>> > >
>>> > > But looks like there is a bug.
>>> > >
>>> > > On Tue, Aug 7, 2012 at 9:21 AM, Stack <[EMAIL PROTECTED]> wrote:
>>> > >
>>> > > > On Tue, Aug 7, 2012 at 9:59 AM, prem yadav <[EMAIL PROTECTED]>

HBASE-6564.

I will try to take a stab on it this weekend.

./zahoor

On Fri, Aug 10, 2012 at 12:47 PM, J Mohamed Zahoor <[EMAIL PROTECTED]> wrote:

> Hi Lars,
>
> Will file it...
>
> ./Zahoor
>
>
> On Fri, Aug 10, 2012 at 12:00 AM, lars hofhansl <[EMAIL PROTECTED]>wrote:
>
>> Hi zahoor,
>>
>> could you file a jira with what you found? It looks like a bug.
>> Thanks.
>>
>>
>> -- Lars
>>
>>
>>
>> ----- Original Message -----
>> From: J Mohamed Zahoor <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Cc:
>> Sent: Thursday, August 9, 2012 2:03 AM
>> Subject: Re: Hbase- Hadoop DFS size not decreased even after deleting a
>> column family
>>
>> Hi
>>
>> I see this atleast in trunk... in TableDeleteFamilyHandler's constructor..
>>
>> public TableDeleteFamilyHandler(byte[] tableName, byte [] familyName,
>>
>>       Server server, final MasterServices masterServices)
>> throwsIOException {
>>
>>     super(EventType.C_M_ADD_FAMILY, tableName, server, masterServices);
>>
>>     HTableDescriptor htd = getTableDescriptor();
>>
>>     this.familyName = hasColumnFamily(htd, familyName);
>>
>>   }
>>
>>
>> The event type is added as C_M_ADD_FAMILY....
>> May be this is the problem and it is there in 0.94.1 also !!!
>>
>> ./zahoor
>>
>>
>> On Wed, Aug 8, 2012 at 5:33 PM, prem yadav <[EMAIL PROTECTED]> wrote:
>>
>> > Hi Stack,
>> > we have got a lot of these in the logs after we deleted the column
>> family.
>> >
>> > org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
>> > family ol does not exist in region <region>
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:4759)
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:2639)
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2065)
>> >         at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1962)
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3428)
>> >         at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
>> >         at
>> >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >         at java.lang.reflect.Method.invoke(Method.java:601)
>> >         at
>> >
>> >
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>> >         at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1386)
>> > 2012-08-05 00:11:37,697 DEBUG
>> org.apache.hadoop.hbase.regionserver.HRegion:
>> > rollbackMemstore rolled back 0 keyvalues from start:0 to end:1
>> >
>> >
>> >
>> >
>> > On Tue, Aug 7, 2012 at 10:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> >
>> > > Looking at TableDeleteFamilyHandler, I see:
>> > >
>> > >   protected void handleTableOperation(List<HRegionInfo> hris) throws
>> > > IOException {
>> > >     // Update table descriptor in HDFS
>> > >     HTableDescriptor htd >> > >
>>  this.masterServices.getMasterFileSystem().deleteColumn(tableName,
>> > > familyName);
>> > >     // Update in-memory descriptor cache
>> > >     this.masterServices.getTableDescriptors().add(htd);
>> > >
>> > > MasterFileSystem.deleteColumn() calls:
>> > >
>> > >     HTableDescriptor htd >> > > this.services.getTableDescriptors().get(tableName);
>> > >     htd.removeFamily(familyName);
>> > >
>> > > I will dig some more.
>> > >
>> > > But looks like there is a bug.
>> > >
>> > > On Tue, Aug 7, 2012 at 9:21 AM, Stack <[EMAIL PROTECTED]> wrote:
>> > >
>> > > > On Tue, Aug 7, 2012 at 9:59 AM, prem yadav <[EMAIL PROTECTED]>
>> > wrote:
>> > > > > Hi Stack,
>> > > > > yes . I can still see the column family.
>> > > > >
>> > > > > we ran the following command too:
>> > > > >
>> > > > > fs -dus /hbase/<table name>/<region hash>/<cf that was deleted>
>> and
>> > it
>> > > > > returned a large value which means that the space has not been
>> >

Hi

Submitted a patch for this.
Should i call for a review for this?

i See lot of other testcases are missing for cases like this in handlers...
i think a Jira would be good for this to track.

./Zahoor
HBase Musings
On 13-Aug-2012, at 11:13 AM, J Mohamed Zahoor <[EMAIL PROTECTED]> wrote:

> HBASE-6564.
>
> I will try to take a stab on it this weekend.
>
> ./zahoor
>
> On Fri, Aug 10, 2012 at 12:47 PM, J Mohamed Zahoor <[EMAIL PROTECTED]> wrote:
> Hi Lars,
>
> Will file it...
> ./Zahoor
>
>
> On Fri, Aug 10, 2012 at 12:00 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> Hi zahoor,
>
> could you file a jira with what you found? It looks like a bug.
> Thanks.
>
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: J Mohamed Zahoor <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc:
> Sent: Thursday, August 9, 2012 2:03 AM
> Subject: Re: Hbase- Hadoop DFS size not decreased even after deleting a column family
>
> Hi
>
> I see this atleast in trunk... in TableDeleteFamilyHandler's constructor..
>
> public TableDeleteFamilyHandler(byte[] tableName, byte [] familyName,
>
>       Server server, final MasterServices masterServices) throwsIOException {
>
>     super(EventType.C_M_ADD_FAMILY, tableName, server, masterServices);
>
>     HTableDescriptor htd = getTableDescriptor();
>
>     this.familyName = hasColumnFamily(htd, familyName);
>
>   }
>
>
> The event type is added as C_M_ADD_FAMILY....
> May be this is the problem and it is there in 0.94.1 also !!!
>
> ./zahoor
>
>
> On Wed, Aug 8, 2012 at 5:33 PM, prem yadav <[EMAIL PROTECTED]> wrote:
>
> > Hi Stack,
> > we have got a lot of these in the logs after we deleted the column family.
> >
> > org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
> > family ol does not exist in region <region>
> >         at
> > org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:4759)
> >         at
> >
> > org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:2639)
> >         at
> >
> > org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2065)
> >         at
> > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1962)
> >         at
> >
> > org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3428)
> >         at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> >         at
> >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:601)
> >         at
> >
> > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> >         at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1386)
> > 2012-08-05 00:11:37,697 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
> > rollbackMemstore rolled back 0 keyvalues from start:0 to end:1
> >
> >
> >
> >
> > On Tue, Aug 7, 2012 at 10:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> > > Looking at TableDeleteFamilyHandler, I see:
> > >
> > >   protected void handleTableOperation(List<HRegionInfo> hris) throws
> > > IOException {
> > >     // Update table descriptor in HDFS
> > >     HTableDescriptor htd > > >       this.masterServices.getMasterFileSystem().deleteColumn(tableName,
> > > familyName);
> > >     // Update in-memory descriptor cache
> > >     this.masterServices.getTableDescriptors().add(htd);
> > >
> > > MasterFileSystem.deleteColumn() calls:
> > >
> > >     HTableDescriptor htd > > > this.services.getTableDescriptors().get(tableName);
> > >     htd.removeFamily(familyName);
> > >
> > > I will dig some more.
> > >
> > > But looks like there is a bug.
> > >
> > > On Tue, Aug 7, 2012 at 9:21 AM, Stack <[EMAIL PROTECTED]> wrote:
> > >
> > > > On Tue, Aug 7, 2012 at 9:59 AM, prem yadav <[EMAIL PROTECTED]>
> > wrote:
> > > > > Hi Stack,
> > > > > yes . I can still see the column family.
> > > > >
> > > > > we ran the following command too: