Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Re: 【region compaction check key-bound or not】


+
Bing Jiang 2013-07-11, 00:40
+
Sergey Shelukhin 2013-07-11, 01:04
+
Ted Yu 2013-07-11, 01:09
Copy link to this message
-
Re: 【region compaction check key-bound or not】
Thanks, Ted and Sergey.
I just want to build a tool to move some regions from one table to another,
so I try to think over the strategy of  Bulkload or snapshot,
but there are more costs in our case.
So could I try to make change in region's compaction, that it adds original
region's key bound in scan?
It is useful to some exception case, for example: the failure of processing
split, which  cannot roll-back.

Store.java:
 StoreFile.Writer compactStore(final Collection<StoreFile> filesToCompact,
                               final boolean majorCompaction, final long
maxId)
{
....
 if (scanner == null) {
          Scan scan = new Scan();
   *       scan.setStartKey(getHRegion().getStartKey());
*
*          scan.setEndKey(getHRegion().getEndKey());*
          scan.setMaxVersions(getFamily().getMaxVersions());
          /* Include deletes, unless we are doing a major compaction */
          scanner = new StoreScanner(this, getScanInfo(), scan, scanners,
            majorCompaction? ScanType.MAJOR_COMPACT :
ScanType.MINOR_COMPACT,
            smallestReadPoint, earliestPutTs);
        }
...
}
Please correct me if mistake.
Thanks.
2013/7/11 Ted Yu <[EMAIL PROTECTED]>

> bq. upgrade to 0.94.6
>
> Nit: upgrade to 0.94.6.1 (or 0.94.9 which was just released)
>
> Cheers
>
> On Wed, Jul 10, 2013 at 6:04 PM, Sergey Shelukhin <[EMAIL PROTECTED]
> >wrote:
>
> > Yeah, this is not going to work... moving storefiles manually between
> > regions is generally not a good idea.
> > Do you want to move entire table into new table? Then the best thing is
> > probably to upgrade to 0.94.6 (rolling restart is supported from 0.94.3)
> > and use HBase snapshots. You can use export table w/o upgrading.
> > If you want to merge into existing table I'm not really sure what is the
> > best way to do that. You might have to rewrite data with target region
> > boundaries, or manually split and merge regions accordingly and run
> > compactions. Then you can disable the table, copy the files and bulk load
> > them into target cluster. See
> > http://hbase.apache.org/book/arch.bulk.load.html
> > Note that bulk load requires boundaries that fit within one region
> > (matching or sub-range). There may be a tool that solves this but I'm not
> > aware of it...
> >
> >
> > On Wed, Jul 10, 2013 at 5:40 PM, Bing Jiang <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Thanks,Sergey.
> > > These days, I want to move a table from one hbase cluster to another
> > hbase
> > > cluster, and there are the same table's schema.
> > > So I want to move a region's storefile to another table's region
> > > corresponding directory, and the key bound of regions are overlapped.
> > > For example:
> > > Cluster   |  Table | Region's key bound
> > > cluster1  |  dat    |  [1ffff,2ffff]
> > > cluster2  |  dat    |  [2bfff,31fff]
> > >
> > > In my opinion , if region files in cluster2 are move into cluster1's
> > > regions, and make compaction(Minor && Major) upon the first region, it
> > will
> > > prune the improper key in (2ffff,31fff].
> > > However I found hbase compaction cannot support that.
> > >
> > > Any idea would be thankful.
> > >
> > >
> > > 2013/7/11 Sergey Shelukhin <[EMAIL PROTECTED]>
> > >
> > > > You should not have to manually take care of region bounds in normal
> > > > circumstances (unless you are reading the file from coprocessor in
> some
> > > > special way, or something like that). Please tell us if you are
> seeing
> > > any
> > > > strange behavior :)
> > > > See HalfStoreFileReader for the code that is used to read the
> > referenced
> > > > file and constrains the keys.
> > > >
> > > > On Wed, Jul 10, 2013 at 4:25 AM, Bing Jiang <
> [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Hi,all
> > > > >
> > > > > If Region process splits, it will make a reference.After executing
> > > child
> > > > > region makes a compaction that absorbs all the reference. And I
> have
> > a
> > > > > question that how to make differences when executes scanner.

Bing Jiang
Tel:(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science