In HBase 0.94, bulkload would always copy the files:

// Move the file if it's on another filesystem
FileSystem srcFs = srcPath.getFileSystem(conf);
if (!srcFs.equals(fs)) {
   LOG.info("File " + srcPath + " on different filesystem than " +
      "destination store - moving to this filesystem.");
    ......

Since fs here is an instance of HFileSystem. This would result in the
load taking long time to complete if the HFiles are in the destination
cluster.

This is fixed in trunk:

FileSystem srcFs = srcPath.getFileSystem(conf);
FileSystem desFs = fs instanceof HFileSystem ?
((HFileSystem)fs).getBackingFs() : fs;
    if (!srcFs.equals(desFs)) {
    ... ...

My question is: should we back port the fix to 0.94?

On Thu, Sep 6, 2012 at 5:30 PM, pig user <[EMAIL PROTECTED]> wrote:
> In HBase 0.94, bulkload would always copy the files:
>
> // Move the file if it's on another filesystem
> FileSystem srcFs = srcPath.getFileSystem(conf);
> if (!srcFs.equals(fs)) {
>    LOG.info("File " + srcPath + " on different filesystem than " +
>       "destination store - moving to this filesystem.");
>     ......
>
> Since fs here is an instance of HFileSystem. This would result in the
> load taking long time to complete if the HFiles are in the destination
> cluster.
>
> This is fixed in trunk:
>
> FileSystem srcFs = srcPath.getFileSystem(conf);
> FileSystem desFs = fs instanceof HFileSystem ?
> ((HFileSystem)fs).getBackingFs() : fs;
>     if (!srcFs.equals(desFs)) {
>     ... ...
>
> My question is: should we back port the fix to 0.94?

Sure, make an issue and backport the patch and we'll commit it.
St.Ack

The fix, HBASE-6529, would be in 0.94.2

Expect 0.94.2 RC0 next week.

Thanks

On Thu, Sep 6, 2012 at 5:30 PM, pig user <[EMAIL PROTECTED]> wrote:

> In HBase 0.94, bulkload would always copy the files:
>
> // Move the file if it's on another filesystem
> FileSystem srcFs = srcPath.getFileSystem(conf);
> if (!srcFs.equals(fs)) {
>    LOG.info("File " + srcPath + " on different filesystem than " +
>       "destination store - moving to this filesystem.");
>     ......
>
> Since fs here is an instance of HFileSystem. This would result in the
> load taking long time to complete if the HFiles are in the destination
> cluster.
>
> This is fixed in trunk:
>
> FileSystem srcFs = srcPath.getFileSystem(conf);
> FileSystem desFs = fs instanceof HFileSystem ?
> ((HFileSystem)fs).getBackingFs() : fs;
>     if (!srcFs.equals(desFs)) {
>     ... ...
>
> My question is: should we back port the fix to 0.94?
>

Thanks Ted. It's great that the fix is already backported.

On Thu, Sep 6, 2012 at 7:19 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> The fix, HBASE-6529, would be in 0.94.2
>
> Expect 0.94.2 RC0 next week.
>
> Thanks
>
> On Thu, Sep 6, 2012 at 5:30 PM, pig user <[EMAIL PROTECTED]> wrote:
>
>> In HBase 0.94, bulkload would always copy the files:
>>
>> // Move the file if it's on another filesystem
>> FileSystem srcFs = srcPath.getFileSystem(conf);
>> if (!srcFs.equals(fs)) {
>>    LOG.info("File " + srcPath + " on different filesystem than " +
>>       "destination store - moving to this filesystem.");
>>     ......
>>
>> Since fs here is an instance of HFileSystem. This would result in the
>> load taking long time to complete if the HFiles are in the destination
>> cluster.
>>
>> This is fixed in trunk:
>>
>> FileSystem srcFs = srcPath.getFileSystem(conf);
>> FileSystem desFs = fs instanceof HFileSystem ?
>> ((HFileSystem)fs).getBackingFs() : fs;
>>     if (!srcFs.equals(desFs)) {
>>     ... ...
>>
>> My question is: should we back port the fix to 0.94?
>>