Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> completebulkload does 'copy' StoreFiles instead of 'cut'


+
M. BagherEsmaeily 2013-09-14, 06:31
+
rajesh babu Chintaguntla 2013-09-14, 10:20
+
M. BagherEsmaeily 2013-09-15, 06:10
+
Ted Yu 2013-09-15, 07:26
+
M. BagherEsmaeily 2013-09-15, 08:09
+
Ted Yu 2013-09-15, 09:42
+
M. BagherEsmaeily 2013-09-15, 09:52
+
Ted Yu 2013-09-15, 14:04
Copy link to this message
-
Re: completebulkload does 'copy' StoreFiles instead of 'cut'
This also happens when the split points of the HFiles to be imported do not match that of the table. In that case the files are split and the split files are written before they get moved into place.

________________________________
 From: Ted Yu <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Sunday, September 15, 2013 7:04 AM
Subject: Re: completebulkload does 'copy' StoreFiles instead of 'cut'
 

In the region server log file, do you see the following ?

      LOG.info("File " + srcPath + " on different filesystem than " +
          "destination store - moving to this filesystem.");
On Sun, Sep 15, 2013 at 2:52 AM, M. BagherEsmaeily <[EMAIL PROTECTED]>wrote:

> Actually that was my fault duplicating the issue.I had no permission to
> delete the second issue. sorry!
>
>
> On Sun, Sep 15, 2013 at 2:12 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > Looks like HBASE-9538 is for the same issue.
> >
> > On Sep 15, 2013, at 1:09 AM, "M. BagherEsmaeily" <[EMAIL PROTECTED]>
> > wrote:
> >
> > > HBASE-9537
> > >
> > >
> > > On Sun, Sep 15, 2013 at 11:56 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > >
> > >> File a JIRA for the issue ?
> > >>
> > >> On Sep 14, 2013, at 11:10 PM, "M. BagherEsmaeily" <
> [EMAIL PROTECTED]
> > >
> > >> wrote:
> > >>
> > >>> Hi rajesh,
> > >>>
> > >>> I use Hbase 0.94.11 and Hadoop 1.2.1. The file system of bulkload
> > >>> output directory and hbase cluster are the same, too.
> > >>>
> > >>> I've also coded a MapReduce job using HFileOutputFormat. When I use
> > >>> LoadIncrementalHFiles
> > >>> to move the output of my job to HBase table, it still copies instead
> of
> > >> cut.
> > >>>
> > >>> Thanks
> > >>>
> > >>>
> > >>> On Sat, Sep 14, 2013 at 2:50 PM, rajesh babu Chintaguntla <
> > >>> [EMAIL PROTECTED]> wrote:
> > >>>
> > >>>> Hi BagherEsmaeily,
> > >>>>
> > >>>>      which version of hbase you are using? Is the file system of
> > >> bulkload
> > >>>> output directory and hbase cluster are same?
> > >>>>
> > >>>>      If you are using hbase older than 0.94.5 version, the
> Storefiles
> > >>>> generated by importtsv are getting copied instead of        moving
> > even
> > >> if
> > >>>> the file system of bulkload output directory and hbase cluster are
> > same.
> > >>>>
> > >>>>      Its a bug and solved in 0.94.5 (HBASE-5498).
> > >>>>
> > >>>> Thanks.
> > >>>> Rajeshbabu
> > >>>>
> > >>>> On Sat, Sep 14, 2013 at 12:01 PM, M. BagherEsmaeily <
> > >> [EMAIL PROTECTED]
> > >>>>> wrote:
> > >>>>
> > >>>>> Hello,
> > >>>>>
> > >>>>> I was using HBase complete bulk load to transfer the output of
> > >> ImportTsv
> > >>>> to
> > >>>>> a table in HBase, and I noticed that it copies the output instead
> of
> > >>>>> cutting. This takes long time for my gigabytes of data.
> > >>>>>
> > >>>>> In HBase documentation (
> > >>>>> http://hbase.apache.org/book/ops_mgt.html#completebulkload) I read
> > >> that
> > >>>>> the
> > >>>>> files would be moved not copied. Can anyone help me with this?
> > >>>>>
> > >>>>> Kind Regards
> > >>
> >
>
+
M. BagherEsmaeily 2013-09-16, 05:50
+
M. BagherEsmaeily 2013-09-16, 06:33