Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Scanner problem after bulk load hfile


Copy link to this message
-
Re: Scanner problem after bulk load hfile
D*mn. Making me feel smart for once. :)
Are you sure there's no old HFile hanging around?

----- Original Message -----
From: Rohit Kelkar <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Cc:
Sent: Tuesday, July 16, 2013 4:39 PM
Subject: Re: Scanner problem after bulk load hfile

Now its working correctly. I had to do a
myTableWriter.appendTrackedTimestampsToMetadata(); after writing my KVs and
before closing the file.

- R
On Tue, Jul 16, 2013 at 6:20 PM, Rohit Kelkar <[EMAIL PROTECTED]> wrote:

> Oh wait. Didn't realise that I had the HbaseAdmin major compact code
> turned on when I tested :(
> It is still not working. Following is the code
>
> StoreFile.Writer myHfileWriter = new StoreFile.WriterBuilder(hbaseConf,
> new CacheConfig(hbaseConf), hdfs,
> HFile.DEFAULT_BLOCKSIZE).withFilePath(myHFilePath).build();
> KeyValue kv = new KeyValue(row.getBytes(), cf.getBytes(),
> keyStr.getBytes(), System.currentTimeMillis(), valueStr.getBytes());
> myHfileWriter.close()
>
> - R
>
>
> On Tue, Jul 16, 2013 at 6:15 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> Looks like the following should be put in RefGuide.
>>
>> Cheers
>>
>> On Tue, Jul 16, 2013 at 3:40 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>>
>> > Hah. Was *just* about to reply with this. The fix in HBASE-8055 is not
>> > strictly necessary.
>> > How did you create your HFiles? See this comment:
>> >
>> https://issues.apache.org/jira/browse/HBASE-8055?focusedCommentId=13600499&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13600499
>> >
>> > -- Lars
>> > ________________________________
>> > From: Jimmy Xiang <[EMAIL PROTECTED]>
>> > To: user <[EMAIL PROTECTED]>
>> > Sent: Tuesday, July 16, 2013 2:41 PM
>> > Subject: Re: Scanner problem after bulk load hfile
>> >
>> >
>> > HBASE-8055 should have fixed it.
>> >
>> >
>> > On Tue, Jul 16, 2013 at 2:33 PM, Rohit Kelkar <[EMAIL PROTECTED]>
>> > wrote:
>> >
>> > > This ( http://pastebin.com/yhx4apCG ) is the error on the region
>> server
>> > > side when execute the following on the shell -
>> > > get 'mytable', 'myrow', 'cf:q'
>> > >
>> > > - R
>> > >
>> > >
>> > >
>> > >
>> > > On Tue, Jul 16, 2013 at 3:28 PM, Jimmy Xiang <[EMAIL PROTECTED]>
>> > wrote:
>> > >
>> > > > Do you see any exception/logging in the region server side?
>> > > >
>> > > >
>> > > > On Tue, Jul 16, 2013 at 1:15 PM, Rohit Kelkar <
>> [EMAIL PROTECTED]>
>> > > > wrote:
>> > > >
>> > > > > Yes. I tried everything from myTable.flushCommits() to
>> > > > > myTable.clearRegionCache() before and after the
>> > > > > LoadIncrementalHFiles.doBulkLoad(). But it doesn't seem to work.
>> This
>> > > is
>> > > > > what I am doing right now to get things moving although I think
>> this
>> > > may
>> > > > > not be the recommended approach -
>> > > > >
>> > > > > HBaseAdmin hbaseAdmin = new HBaseAdmin(hbaseConf);
>> > > > > hbaseAdmin.majorCompact(myTableName.getBytes());
>> > > > > myTable.close();
>> > > > > hbaseAdmin.close();
>> > > > >
>> > > > > - R
>> > > > >
>> > > > >
>> > > > > On Mon, Jul 15, 2013 at 9:14 AM, Amit Sela <[EMAIL PROTECTED]>
>> > > wrote:
>> > > > >
>> > > > > > Well, I know it's kind of voodoo but try it once before
>> pre-split
>> > and
>> > > > > once
>> > > > > > after. Worked for me.
>> > > > > >
>> > > > > >
>> > > > > > On Mon, Jul 15, 2013 at 7:27 AM, Rohit Kelkar <
>> > [EMAIL PROTECTED]
>> > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Thanks Amit, I am also using 0.94.2 . I am also pre-splitting
>> > and I
>> > > > > tried
>> > > > > > > the table.clearRegionCache() but still doesn't work.
>> > > > > > >
>> > > > > > > - R
>> > > > > > >
>> > > > > > >
>> > > > > > > On Sun, Jul 14, 2013 at 3:45 AM, Amit Sela <
>> [EMAIL PROTECTED]>
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > If new regions are created during the bulk load (are you
>> > > > > pre-splitting
>> > > > > > > ?),
>> > > > > > > > maybe try myTable.clearRegionCache() after the bulk load (or