Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Scanner problem after bulk load hfile


Copy link to this message
-
Re: Scanner problem after bulk load hfile
D*mn. Making me feel smart for once. :)
Are you sure there's no old HFile hanging around?

----- Original Message -----
From: Rohit Kelkar <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Cc:
Sent: Tuesday, July 16, 2013 4:39 PM
Subject: Re: Scanner problem after bulk load hfile

Now its working correctly. I had to do a
myTableWriter.appendTrackedTimestampsToMetadata(); after writing my KVs and
before closing the file.

- R
On Tue, Jul 16, 2013 at 6:20 PM, Rohit Kelkar <[EMAIL PROTECTED]> wrote:

> Oh wait. Didn't realise that I had the HbaseAdmin major compact code
> turned on when I tested :(
> It is still not working. Following is the code
>
> StoreFile.Writer myHfileWriter = new StoreFile.WriterBuilder(hbaseConf,
> new CacheConfig(hbaseConf), hdfs,
> HFile.DEFAULT_BLOCKSIZE).withFilePath(myHFilePath).build();
> KeyValue kv = new KeyValue(row.getBytes(), cf.getBytes(),
> keyStr.getBytes(), System.currentTimeMillis(), valueStr.getBytes());
> myHfileWriter.close()
>
> - R
>
>
> On Tue, Jul 16, 2013 at 6:15 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> Looks like the following should be put in RefGuide.
>>
>> Cheers
>>
>> On Tue, Jul 16, 2013 at 3:40 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>>
>> > Hah. Was *just* about to reply with this. The fix in HBASE-8055 is not
>> > strictly necessary.
>> > How did you create your HFiles? See this comment:
>> >
>> https://issues.apache.org/jira/browse/HBASE-8055?focusedCommentId=13600499&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13600499
>> >
>> > -- Lars
>> > ________________________________
>> > From: Jimmy Xiang <[EMAIL PROTECTED]>
>> > To: user <[EMAIL PROTECTED]>
>> > Sent: Tuesday, July 16, 2013 2:41 PM
>> > Subject: Re: Scanner problem after bulk load hfile
>> >
>> >
>> > HBASE-8055 should have fixed it.
>> >
>> >
>> > On Tue, Jul 16, 2013 at 2:33 PM, Rohit Kelkar <[EMAIL PROTECTED]>
>> > wrote:
>> >
>> > > This ( http://pastebin.com/yhx4apCG ) is the error on the region
>> server
>> > > side when execute the following on the shell -
>> > > get 'mytable', 'myrow', 'cf:q'
>> > >
>> > > - R
>> > >
>> > >
>> > >
>> > >
>> > > On Tue, Jul 16, 2013 at 3:28 PM, Jimmy Xiang <[EMAIL PROTECTED]>
>> > wrote:
>> > >
>> > > > Do you see any exception/logging in the region server side?
>> > > >
>> > > >
>> > > > On Tue, Jul 16, 2013 at 1:15 PM, Rohit Kelkar <
>> [EMAIL PROTECTED]>
>> > > > wrote:
>> > > >
>> > > > > Yes. I tried everything from myTable.flushCommits() to
>> > > > > myTable.clearRegionCache() before and after the
>> > > > > LoadIncrementalHFiles.doBulkLoad(). But it doesn't seem to work.
>> This
>> > > is
>> > > > > what I am doing right now to get things moving although I think
>> this
>> > > may
>> > > > > not be the recommended approach -
>> > > > >
>> > > > > HBaseAdmin hbaseAdmin = new HBaseAdmin(hbaseConf);
>> > > > > hbaseAdmin.majorCompact(myTableName.getBytes());
>> > > > > myTable.close();
>> > > > > hbaseAdmin.close();
>> > > > >
>> > > > > - R
>> > > > >
>> > > > >
>> > > > > On Mon, Jul 15, 2013 at 9:14 AM, Amit Sela <[EMAIL PROTECTED]>
>> > > wrote:
>> > > > >
>> > > > > > Well, I know it's kind of voodoo but try it once before
>> pre-split
>> > and
>> > > > > once
>> > > > > > after. Worked for me.
>> > > > > >
>> > > > > >
>> > > > > > On Mon, Jul 15, 2013 at 7:27 AM, Rohit Kelkar <
>> > [EMAIL PROTECTED]
>> > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Thanks Amit, I am also using 0.94.2 . I am also pre-splitting
>> > and I
>> > > > > tried
>> > > > > > > the table.clearRegionCache() but still doesn't work.
>> > > > > > >
>> > > > > > > - R
>> > > > > > >
>> > > > > > >
>> > > > > > > On Sun, Jul 14, 2013 at 3:45 AM, Amit Sela <
>> [EMAIL PROTECTED]>
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > If new regions are created during the bulk load (are you
>> > > > > pre-splitting
>> > > > > > > ?),
>> > > > > > > > maybe try myTable.clearRegionCache() after the bulk load (or
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB