Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - HBase Random Read latency > 100ms


Copy link to this message
-
Re: HBase Random Read latency > 100ms
Ramu M S 2013-10-07, 11:26
Lars, Bharath,

Compression is disabled for the table. This was not intended from the
evaluation.
I forgot to mention that during table creation. I will enable snappy and do
major compaction again.

Please suggest other options to try out and also suggestions for the
previous questions.

Thanks,
Ramu
On Mon, Oct 7, 2013 at 6:35 PM, Ramu M S <[EMAIL PROTECTED]> wrote:

> Bharath,
>
> I was about to report this. Yes indeed there is too much of GC time.
> Just verified the GC time using Cloudera Manager statistics(Every minute
> update).
>
> For each Region Server,
>  - During Read: Graph shows 2s constant.
>  - During Compaction: Graph starts with 7s and goes as high as 20s during
> end.
>
> Few more questions,
> 1. For the current evaluation, since the reads are completely random and I
> don't expect to read same data again can I set the Heap to the default 1 GB
> ?
>
> 2. Can I completely turn off BLOCK CACHE for this table?
>     http://hbase.apache.org/book/regionserver.arch.html recommends that
> for Randm reads.
>
> 3. But in the next phase of evaluation, We are interested to use HBase as
> In-memory KV DB by having the latest data in RAM (To the tune of around 128
> GB in each RS, we are setting up 50-100 Node Cluster). I am very curious to
> hear any suggestions in this regard.
>
> Regards,
> Ramu
>
>
> On Mon, Oct 7, 2013 at 5:50 PM, Bharath Vissapragada <
> [EMAIL PROTECTED]> wrote:
>
>> Hi Ramu,
>>
>> Thanks for reporting the results back. Just curious if you are hitting any
>> big GC pauses due to block cache churn on such large heap. Do you see it ?
>>
>> - Bharath
>>
>>
>> On Mon, Oct 7, 2013 at 1:42 PM, Ramu M S <[EMAIL PROTECTED]> wrote:
>>
>> > Lars,
>> >
>> > After changing the BLOCKSIZE to 16KB, the latency has reduced a little.
>> Now
>> > the average is around 75ms.
>> > Overall throughput (I am using 40 Clients to fetch records) is around 1K
>> > OPS.
>> >
>> > After compaction hdfsBlocksLocalityIndex is 91,88,78,90,99,82,94,97 in
>> my 8
>> > RS respectively.
>> >
>> > Thanks,
>> > Ramu
>> >
>> >
>> > On Mon, Oct 7, 2013 at 3:51 PM, Ramu M S <[EMAIL PROTECTED]> wrote:
>> >
>> > > Thanks Lars.
>> > >
>> > > I have changed the BLOCKSIZE to 16KB and triggered a major
>> compaction. I
>> > > will report my results once it is done.
>> > >
>> > > - Ramu
>> > >
>> > >
>> > > On Mon, Oct 7, 2013 at 3:21 PM, lars hofhansl <[EMAIL PROTECTED]>
>> wrote:
>> > >
>> > >> First of: 128gb heap per RegionServer. Wow.I'd be interested to hear
>> > your
>> > >> experience with such a large heap for your RS. It's definitely big
>> > enough.
>> > >>
>> > >>
>> > >> It's interesting hat 100gb do fit into the aggregate cache (of
>> 8x32gb),
>> > >> while 1.8tb do not.
>> > >> Looks like ~70% of the read request would need to bring in a 64kb
>> block
>> > >> in order to read 724 bytes.
>> > >>
>> > >> Should that take 100ms? No. Something's still amiss.
>> > >>
>> > >> Smaller blocks might help (you'd need to bring in 4, 8, or maybe 16k
>> to
>> > >> read the small row). You would need to issue a major compaction for
>> > that to
>> > >> take effect.
>> > >> Maybe try 16k blocks. If that speeds up your random gets we know
>> where
>> > to
>> > >> look next... At the disk IO.
>> > >>
>> > >>
>> > >> -- Lars
>> > >>
>> > >>
>> > >>
>> > >> ________________________________
>> > >>  From: Ramu M S <[EMAIL PROTECTED]>
>> > >> To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
>> > >> Sent: Sunday, October 6, 2013 11:05 PM
>> > >> Subject: Re: HBase Random Read latency > 100ms
>> > >>
>> > >>
>> > >> Lars,
>> > >>
>> > >> In one of your old posts, you had mentioned that lowering the
>> BLOCKSIZE
>> > is
>> > >> good for random reads (of course with increased size for Block
>> Indexes).
>> > >>
>> > >> Post is at
>> > http://grokbase.com/t/hbase/user/11bat80x7m/row-get-very-slow
>> > >>
>> > >> Will that help in my tests? Should I give it a try? If I alter my
>> table,
>> > >> should I trigger a major compaction again for this to take effect?