Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Get the list of store/store files for a region via HBase API


Copy link to this message
-
Re: Get the list of store/store files for a region via HBase API

With respect to #2, it's more than that.  StoreFiles get written on
MemStore flushes but also re-written whenever compactions happen.  So
regardless of where you get this information, you have to expect that this
answer is going to change.

On 5/16/12 12:22 PM, "Chen Song" <[EMAIL PROTECTED]> wrote:

>In HBase API, there are classes defined on each level of structs. For
>example, HRegionInterface, HRegionInfo (HRegion), Store, StoreFile. I am
>not sure why there doesn't seem to have a clear way to traverse this
>hierarchical structure.
>
>I can think of some downsides of using HDFS to get such information.
>
>1. The user has to rebuild and maintain a similar hierarchy of struct
>classes on their end from the HDFS structure. For simple use case such as
>getting counts, it is probably OK but it could enforce the user to
>replicate the similar amount of work (which has been done on HBase API)
>for
>use cases that need more complicated meta information.
>2. If a region is removed from meta store, there exists a timing between
>the time it is removed from meta store and that it is physically removed
>from HDFS. That said, it is not reliable.
>
>Chen
>
>On Tue, May 15, 2012 at 10:06 PM, Doug Meil
><[EMAIL PROTECTED]>wrote:
>
>>
>> You're just doing a directory listing in HDFS to get this information.
>> That's a pretty lightweight operation (I.e., as opposed to transferring
>> the contents of all the StoreFiles, etc.)
>>
>> If you want to go all the way down to the StoreFile it's the only way
>>I'm
>> aware of at this time.
>>
>>
>>
>>
>> On 5/15/12 5:37 PM, "Chen Song" <[EMAIL PROTECTED]> wrote:
>>
>> >Thanks Doug, that should work as the hierarchy is explicitly reflected
>>on
>> >HDFS. But is this the preferred way to do such table/region/storefile
>> >traversal?
>> >
>> >I would like to avoid hitting to HDFS directly if possible as these
>>pieces
>> >of meta information is within HBase world.
>> >
>> >Thanks
>> >Chen
>> >
>> >On Tue, May 15, 2012 at 5:23 PM, Doug Meil
>> ><[EMAIL PROTECTED]>wrote:
>> >
>> >>
>> >> You can get the Table->Region->StoreFile information via HDFS.  That
>>is
>> >> described here in the RefGuide:
>> >>
>> >> http://hbase.apache.org/book.html#trouble.namenode
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On 5/15/12 5:09 PM, "Chen Song" <[EMAIL PROTECTED]> wrote:
>> >>
>> >> >I am new to HBase and started working on a project which needs meta
>> >> >information on HBase regions for a table. The version of HBase I am
>> >>using
>> >> >0.90.4.
>> >> >
>> >> >The use case is very simple.
>> >> >
>> >> >First, I want to get all regions for a table, which I can achieve
>>using
>> >> >the
>> >> >API call below.
>> >> >
>> >> >    HTable table = new HTable(conf, tableName);
>> >> >
>> >> >    Map<HRegionInfo, HServerAddress> regions >>table.getRegionsInfo();
>> >> >
>> >> >
>> >> >Second, I want to get the list of stores (and then store files) for
>> >>each
>> >> >region. This is where I stuck as I could not find a way to do it by
>> >> >searching in the API. It seems that HRegionInterface started
>>supporting
>> >> >API
>> >> >call to retrive the list of store files since 0.95-SNAPSHOT but I
>>don't
>> >> >want to upgrade my HBase version. Below is how to get the
>>corresponding
>> >> >HResionInterface.
>> >> >
>> >> >    HConnection connection = table.getConnection();
>> >> >
>> >> >    HRegionInterface regionInterface >> >> >connection.getHRegionConnection(regionAddress);
>> >> >
>> >> >The series of objects I would like to get in sequence is like:
>> >>HRegionInfo
>> >> >-> HRegion -> Store/StoreFile/HFile. If some can help me with it,
>>that
>> >> >would be great.
>> >> >
>> >> >Thanks very much,
>> >> >
>> >> >Chen
>> >>
>> >>
>> >>
>> >
>> >
>> >--
>> >Chen Song
>> >Mobile: 518-445-5096
>>
>>
>>
>
>
>--
>Chen Song
>Mobile: 518-445-5096