Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Questions about HBase


+
Pankaj Gupta 2013-06-05, 02:15
+
Ted Yu 2013-06-05, 03:44
+
ramkrishna vasudevan 2013-06-05, 03:14
+
Ted Yu 2013-06-05, 04:29
+
Anoop John 2013-06-05, 05:26
+
Pankaj Gupta 2013-06-05, 05:09
+
Asaf Mesika 2013-06-05, 05:27
+
ramkrishna vasudevan 2013-06-05, 05:43
+
Asaf Mesika 2013-06-05, 05:52
+
ramkrishna vasudevan 2013-06-05, 06:05
+
Pankaj Gupta 2013-06-05, 06:16
+
Pankaj Gupta 2013-06-05, 06:26
Copy link to this message
-
Re: Questions about HBase
ramkrishna vasudevan 2013-06-05, 07:07
I get your point Pankaj.
Going thro the code to confirm it
    // Data index. We also read statistics about the block index written
after
    // the root level.
    dataBlockIndexReader.readMultiLevelIndexRoot(
        blockIter.nextBlockWithBlockType(BlockType.ROOT_INDEX),
        trailer.getDataIndexCount());

    // Meta index.
    metaBlockIndexReader.readRootIndex(
        blockIter.nextBlockWithBlockType(BlockType.ROOT_INDEX),
        trailer.getMetaIndexCount());

We read the root level of the multilevel index and the actual root index.
So as and when when we need new index blocks we will be hitting the disk
and your observation is correct.  Sorry if i had confused you in this.
The new version of HFile was mainly to address the concern in the previous
versoin where the entire indices was in memory.  The version V2 addressed
that concern like having the root level (something like metadata of the
indices) and from there you should be able to get new index blocks.
But there are chances that if you region size is small you may have only
one level and the entire thing may be in memory.

Regards
Ram
On Wed, Jun 5, 2013 at 11:56 AM, Pankaj Gupta <[EMAIL PROTECTED]> wrote:

> Sorry, forgot to mention that I added the log statements to the method
> readBlock in HFileReaderV2.java. I'm on hbase 0.94.2.
>
>
> On Tue, Jun 4, 2013 at 11:16 PM, Pankaj Gupta <[EMAIL PROTECTED]>
> wrote:
>
> > Some context on how I observed bloom filters being loaded constantly. I
> > added the following logging statements to HFileReaderV2.java:
> > }
> >         if (!useLock) {
> >           // check cache again with lock
> >           useLock = true;
> >           continue;
> >         }
> >
> >         // Load block from filesystem.
> >         long startTimeNs = System.nanoTime();
> >         HFileBlock hfileBlock > > fsBlockReader.readBlockData(dataBlockOffset,
> >             onDiskBlockSize, -1, pread);
> >         hfileBlock = dataBlockEncoder.diskToCacheFormat(hfileBlock,
> >             isCompaction);
> >         validateBlockType(hfileBlock, expectedBlockType);
> >         passSchemaMetricsTo(hfileBlock);
> >         BlockCategory blockCategory > > hfileBlock.getBlockType().getCategory();
> >
> > // My logging statements ---->
> >         if(blockCategory == BlockCategory.INDEX) {
> >           LOG.info("index block miss, reading from disk " + cacheKey);
> >         } else if (blockCategory == BlockCategory.BLOOM) {
> >           LOG.info("bloom block miss, reading from disk " + cacheKey);
> >         } else {
> >           LOG.info("block miss other than index or bloom, reading from
> > disk " + cacheKey);
> >         }
> > //-------------->
> >         final long delta = System.nanoTime() - startTimeNs;
> >         HFile.offerReadLatency(delta, pread);
> >         getSchemaMetrics().updateOnCacheMiss(blockCategory, isCompaction,
> > delta);
> >
> >         // Cache the block if necessary
> >         if (cacheBlock && cacheConf.shouldCacheBlockOnRead(
> >             hfileBlock.getBlockType().getCategory())) {
> >           cacheConf.getBlockCache().cacheBlock(cacheKey, hfileBlock,
> >               cacheConf.isInMemory());
> >         }
> >
> >         if (hfileBlock.getBlockType() == BlockType.DATA) {
> >           HFile.dataBlockReadCnt.incrementAndGet();
> >         }
> >
> > With these in place I saw the following statements in log:
> > 2013-06-05 01:04:55,281 INFO
> > org.apache.hadoop.hbase.io.hfile.HFileReaderV2: index block miss, reading
> > from disk 11958ab7a4a1492e853743b02e1bd7b1_30361506
> > 2013-06-05 01:05:00,579 INFO
> > org.apache.hadoop.hbase.io.hfile.HFileReaderV2: index block miss, reading
> > from disk 52cded0c399b48fdbccd8b3d4e25502f_28779560
> > 2013-06-05 01:07:41,335 INFO
> > org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
> > from disk 11958ab7a4a1492e853743b02e1bd7b1_4199735
> > 2013-06-05 01:08:58,460 INFO
> > org.apache.hadoop.hbase.io.hfile.HFileReaderV2: bloom block miss, reading
+
Anoop John 2013-06-05, 08:24
+
Pankaj Gupta 2013-06-06, 02:52
+
ramkrishna vasudevan 2013-06-06, 03:15
+
Anoop John 2013-06-06, 05:38
+
Pankaj Gupta 2013-06-05, 06:06