Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Re: 【region compaction check key-bound or not】


+
Bing Jiang 2013-07-11, 00:40
+
Sergey Shelukhin 2013-07-11, 01:04
+
Ted Yu 2013-07-11, 01:09
Copy link to this message
-
Re: 【region compaction check key-bound or not】
Thanks, Ted and Sergey.
I just want to build a tool to move some regions from one table to another,
so I try to think over the strategy of  Bulkload or snapshot,
but there are more costs in our case.
So could I try to make change in region's compaction, that it adds original
region's key bound in scan?
It is useful to some exception case, for example: the failure of processing
split, which  cannot roll-back.

Store.java:
 StoreFile.Writer compactStore(final Collection<StoreFile> filesToCompact,
                               final boolean majorCompaction, final long
maxId)
{
....
 if (scanner == null) {
          Scan scan = new Scan();
   *       scan.setStartKey(getHRegion().getStartKey());
*
*          scan.setEndKey(getHRegion().getEndKey());*
          scan.setMaxVersions(getFamily().getMaxVersions());
          /* Include deletes, unless we are doing a major compaction */
          scanner = new StoreScanner(this, getScanInfo(), scan, scanners,
            majorCompaction? ScanType.MAJOR_COMPACT :
ScanType.MINOR_COMPACT,
            smallestReadPoint, earliestPutTs);
        }
...
}
Please correct me if mistake.
Thanks.
2013/7/11 Ted Yu <[EMAIL PROTECTED]>

> bq. upgrade to 0.94.6
>
> Nit: upgrade to 0.94.6.1 (or 0.94.9 which was just released)
>
> Cheers
>
> On Wed, Jul 10, 2013 at 6:04 PM, Sergey Shelukhin <[EMAIL PROTECTED]
> >wrote:
>
> > Yeah, this is not going to work... moving storefiles manually between
> > regions is generally not a good idea.
> > Do you want to move entire table into new table? Then the best thing is
> > probably to upgrade to 0.94.6 (rolling restart is supported from 0.94.3)
> > and use HBase snapshots. You can use export table w/o upgrading.
> > If you want to merge into existing table I'm not really sure what is the
> > best way to do that. You might have to rewrite data with target region
> > boundaries, or manually split and merge regions accordingly and run
> > compactions. Then you can disable the table, copy the files and bulk load
> > them into target cluster. See
> > http://hbase.apache.org/book/arch.bulk.load.html
> > Note that bulk load requires boundaries that fit within one region
> > (matching or sub-range). There may be a tool that solves this but I'm not
> > aware of it...
> >
> >
> > On Wed, Jul 10, 2013 at 5:40 PM, Bing Jiang <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Thanks,Sergey.
> > > These days, I want to move a table from one hbase cluster to another
> > hbase
> > > cluster, and there are the same table's schema.
> > > So I want to move a region's storefile to another table's region
> > > corresponding directory, and the key bound of regions are overlapped.
> > > For example:
> > > Cluster   |  Table | Region's key bound
> > > cluster1  |  dat    |  [1ffff,2ffff]
> > > cluster2  |  dat    |  [2bfff,31fff]
> > >
> > > In my opinion , if region files in cluster2 are move into cluster1's
> > > regions, and make compaction(Minor && Major) upon the first region, it
> > will
> > > prune the improper key in (2ffff,31fff].
> > > However I found hbase compaction cannot support that.
> > >
> > > Any idea would be thankful.
> > >
> > >
> > > 2013/7/11 Sergey Shelukhin <[EMAIL PROTECTED]>
> > >
> > > > You should not have to manually take care of region bounds in normal
> > > > circumstances (unless you are reading the file from coprocessor in
> some
> > > > special way, or something like that). Please tell us if you are
> seeing
> > > any
> > > > strange behavior :)
> > > > See HalfStoreFileReader for the code that is used to read the
> > referenced
> > > > file and constrains the keys.
> > > >
> > > > On Wed, Jul 10, 2013 at 4:25 AM, Bing Jiang <
> [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Hi,all
> > > > >
> > > > > If Region process splits, it will make a reference.After executing
> > > child
> > > > > region makes a compaction that absorbs all the reference. And I
> have
> > a
> > > > > question that how to make differences when executes scanner.

Bing Jiang
Tel:(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB