Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - Unnecessary file copying during the bulkload: should we backport the fix in 0.96?


Copy link to this message
-
Re: Unnecessary file copying during the bulkload: should we backport the fix in 0.96?
Ted Yu 2012-09-07, 02:19
The fix, HBASE-6529, would be in 0.94.2

Expect 0.94.2 RC0 next week.

Thanks

On Thu, Sep 6, 2012 at 5:30 PM, pig user <[EMAIL PROTECTED]> wrote:

> In HBase 0.94, bulkload would always copy the files:
>
> // Move the file if it's on another filesystem
> FileSystem srcFs = srcPath.getFileSystem(conf);
> if (!srcFs.equals(fs)) {
>    LOG.info("File " + srcPath + " on different filesystem than " +
>       "destination store - moving to this filesystem.");
>     ......
>
> Since fs here is an instance of HFileSystem. This would result in the
> load taking long time to complete if the HFiles are in the destination
> cluster.
>
> This is fixed in trunk:
>
> FileSystem srcFs = srcPath.getFileSystem(conf);
> FileSystem desFs = fs instanceof HFileSystem ?
> ((HFileSystem)fs).getBackingFs() : fs;
>     if (!srcFs.equals(desFs)) {
>     ... ...
>
> My question is: should we back port the fix to 0.94?
>