I just have 5 stress client threads writing timeseries data. What I see is
after few mts HBaseClient slows down and starts to take 4 secs. Once I kill
the client and restart it stays at sustainable rate for about 2 mts and
then again it slows down. I am wondering if there is something I should be
doing on the HBaseclient side? All the request are similar in terms of data.

It's one of those "it depends" answers.

See this firstŠ

http://hbase.apache.org/book.html#perf.writing

Š Additionally, one thing to understand is where you are writing data.
Either keep track of the requests per RS over the period (e.g., the web
interface), or you can also track it on the client side with...

http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#
getRegionLocation%28byte[],%20boolean%29
Š to know if you are continually hitting the same RS or spreading the load.

On 10/9/12 1:27 PM, "Mohit Anchlia" <[EMAIL PROTECTED]> wrote:

>I just have 5 stress client threads writing timeseries data. What I see is
>after few mts HBaseClient slows down and starts to take 4 secs. Once I
>kill
>the client and restart it stays at sustainable rate for about 2 mts and
>then again it slows down. I am wondering if there is something I should be
>doing on the HBaseclient side? All the request are similar in terms of
>data.

I am using HTableInterface as a pool but I don't see any setautoflush
method. I am using 0.92.1 jar.

Also, how can I see if RS is getting overloaded? I looked at the UI and I
don't see anything obvious:

equestsPerSecond=0, numberOfOnlineRegions=1, numberOfStores=1,
numberOfStorefiles=1, storefileIndexSizeMB=0, rootIndexSizeKB=1,
totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, memstoreSizeMB=27,
readRequestsCount=126, writeRequestsCount=96157, compactionQueueSize=0,
flushQueueSize=0, usedHeapMB=44, maxHeapMB=3976, blockCacheSizeMB=8.79,
blockCacheFreeMB=985.34, blockCacheCount=11, blockCacheHitCount=23,
blockCacheMissCount=28, blockCacheEvictedCount=0, blockCacheHitRatio=45%,
blockCacheHitCachingRatio=67%, hdfsBlocksLocalityIndex=100

On Tue, Oct 9, 2012 at 10:32 AM, Doug Meil <[EMAIL PROTECTED]>wrote:

>
> It's one of those "it depends" answers.
>
> See this firstŠ
>
> http://hbase.apache.org/book.html#perf.writing
>
> Š Additionally, one thing to understand is where you are writing data.
> Either keep track of the requests per RS over the period (e.g., the web
> interface), or you can also track it on the client side with...
>
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#
> getRegionLocation%28byte[],%20boolean%29
>
>
> Š to know if you are continually hitting the same RS or spreading the load.
>
>
>
> On 10/9/12 1:27 PM, "Mohit Anchlia" <[EMAIL PROTECTED]> wrote:
>
> >I just have 5 stress client threads writing timeseries data. What I see is
> >after few mts HBaseClient slows down and starts to take 4 secs. Once I
> >kill
> >the client and restart it stays at sustainable rate for about 2 mts and
> >then again it slows down. I am wondering if there is something I should be
> >doing on the HBaseclient side? All the request are similar in terms of
> >data.
>
>
>

So you're running on a single regionserver?
On 10/9/12 1:44 PM, "Mohit Anchlia" <[EMAIL PROTECTED]> wrote:

>I am using HTableInterface as a pool but I don't see any setautoflush
>method. I am using 0.92.1 jar.
>
>Also, how can I see if RS is getting overloaded? I looked at the UI and I
>don't see anything obvious:
>
>equestsPerSecond=0, numberOfOnlineRegions=1, numberOfStores=1,
>numberOfStorefiles=1, storefileIndexSizeMB=0, rootIndexSizeKB=1,
>totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, memstoreSizeMB=27,
>readRequestsCount=126, writeRequestsCount=96157, compactionQueueSize=0,
>flushQueueSize=0, usedHeapMB=44, maxHeapMB=3976, blockCacheSizeMB=8.79,
>blockCacheFreeMB=985.34, blockCacheCount=11, blockCacheHitCount=23,
>blockCacheMissCount=28, blockCacheEvictedCount=0, blockCacheHitRatio=45%,
>blockCacheHitCachingRatio=67%, hdfsBlocksLocalityIndex=100
>
>On Tue, Oct 9, 2012 at 10:32 AM, Doug Meil
><[EMAIL PROTECTED]>wrote:
>
>>
>> It's one of those "it depends" answers.
>>
>> See this firstŠ
>>
>> http://hbase.apache.org/book.html#perf.writing
>>
>> Š Additionally, one thing to understand is where you are writing data.
>> Either keep track of the requests per RS over the period (e.g., the web
>> interface), or you can also track it on the client side with...
>>
>>
>>http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.htm
>>l#
>> getRegionLocation%28byte[],%20boolean%29
>>
>>
>> Š to know if you are continually hitting the same RS or spreading the
>>load.
>>
>>
>>
>> On 10/9/12 1:27 PM, "Mohit Anchlia" <[EMAIL PROTECTED]> wrote:
>>
>> >I just have 5 stress client threads writing timeseries data. What I
>>see is
>> >after few mts HBaseClient slows down and starts to take 4 secs. Once I
>> >kill
>> >the client and restart it stays at sustainable rate for about 2 mts and
>> >then again it slows down. I am wondering if there is something I
>>should be
>> >doing on the HBaseclient side? All the request are similar in terms of
>> >data.
>>
>>
>>

There are 2 CF on 2 separate region server. And yes, I have not pre-split
the regions as I was told that we should let HBase handle that
automatically.

Is there a way to set autoflush when using HTableDescriptor?

On Tue, Oct 9, 2012 at 10:50 AM, Doug Meil <[EMAIL PROTECTED]>wrote:

>
> So you're running on a single regionserver?
>
>
>
>
> On 10/9/12 1:44 PM, "Mohit Anchlia" <[EMAIL PROTECTED]> wrote:
>
> >I am using HTableInterface as a pool but I don't see any setautoflush
> >method. I am using 0.92.1 jar.
> >
> >Also, how can I see if RS is getting overloaded? I looked at the UI and I
> >don't see anything obvious:
> >
> >equestsPerSecond=0, numberOfOnlineRegions=1, numberOfStores=1,
> >numberOfStorefiles=1, storefileIndexSizeMB=0, rootIndexSizeKB=1,
> >totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, memstoreSizeMB=27,
> >readRequestsCount=126, writeRequestsCount=96157, compactionQueueSize=0,
> >flushQueueSize=0, usedHeapMB=44, maxHeapMB=3976, blockCacheSizeMB=8.79,
> >blockCacheFreeMB=985.34, blockCacheCount=11, blockCacheHitCount=23,
> >blockCacheMissCount=28, blockCacheEvictedCount=0, blockCacheHitRatio=45%,
> >blockCacheHitCachingRatio=67%, hdfsBlocksLocalityIndex=100
> >
> >On Tue, Oct 9, 2012 at 10:32 AM, Doug Meil
> ><[EMAIL PROTECTED]>wrote:
> >
> >>
> >> It's one of those "it depends" answers.
> >>
> >> See this firstŠ
> >>
> >> http://hbase.apache.org/book.html#perf.writing
> >>
> >> Š Additionally, one thing to understand is where you are writing data.
> >> Either keep track of the requests per RS over the period (e.g., the web
> >> interface), or you can also track it on the client side with...
> >>
> >>
> >>
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.htm
> >>l#
> >> getRegionLocation%28byte[],%20boolean%29
> >>
> >>
> >> Š to know if you are continually hitting the same RS or spreading the
> >>load.
> >>
> >>
> >>
> >> On 10/9/12 1:27 PM, "Mohit Anchlia" <[EMAIL PROTECTED]> wrote:
> >>
> >> >I just have 5 stress client threads writing timeseries data. What I
> >>see is
> >> >after few mts HBaseClient slows down and starts to take 4 secs. Once I
> >> >kill
> >> >the client and restart it stays at sustainable rate for about 2 mts and
> >> >then again it slows down. I am wondering if there is something I
> >>should be
> >> >doing on the HBaseclient side? All the request are similar in terms of
> >> >data.
> >>
> >>
> >>
>
>
>

It looks as if RS is able to take the load but at some point memory buffer
on the server is full and it slows everything down.

Some interseting points I am seeing is memstore size of 50MB,
fssynclatency_num_ops= 300k, fswritelatency=180k

On Tue, Oct 9, 2012 at 11:03 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:

> There are 2 CF on 2 separate region server. And yes, I have not pre-split
> the regions as I was told that we should let HBase handle that
> automatically.
>
> Is there a way to set autoflush when using HTableDescriptor?
>
>  On Tue, Oct 9, 2012 at 10:50 AM, Doug Meil <[EMAIL PROTECTED]
> > wrote:
>
>>
>> So you're running on a single regionserver?
>>
>>
>>
>>
>> On 10/9/12 1:44 PM, "Mohit Anchlia" <[EMAIL PROTECTED]> wrote:
>>
>> >I am using HTableInterface as a pool but I don't see any setautoflush
>> >method. I am using 0.92.1 jar.
>> >
>> >Also, how can I see if RS is getting overloaded? I looked at the UI and I
>> >don't see anything obvious:
>> >
>> >equestsPerSecond=0, numberOfOnlineRegions=1, numberOfStores=1,
>> >numberOfStorefiles=1, storefileIndexSizeMB=0, rootIndexSizeKB=1,
>> >totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, memstoreSizeMB=27,
>> >readRequestsCount=126, writeRequestsCount=96157, compactionQueueSize=0,
>> >flushQueueSize=0, usedHeapMB=44, maxHeapMB=3976, blockCacheSizeMB=8.79,
>> >blockCacheFreeMB=985.34, blockCacheCount=11, blockCacheHitCount=23,
>> >blockCacheMissCount=28, blockCacheEvictedCount=0, blockCacheHitRatio=45%,
>> >blockCacheHitCachingRatio=67%, hdfsBlocksLocalityIndex=100
>> >
>> >On Tue, Oct 9, 2012 at 10:32 AM, Doug Meil
>> ><[EMAIL PROTECTED]>wrote:
>> >
>> >>
>> >> It's one of those "it depends" answers.
>> >>
>> >> See this firstŠ
>> >>
>> >> http://hbase.apache.org/book.html#perf.writing
>> >>
>> >> Š Additionally, one thing to understand is where you are writing data.
>> >> Either keep track of the requests per RS over the period (e.g., the web
>> >> interface), or you can also track it on the client side with...
>> >>
>> >>
>> >>
>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.htm
>> >>l#
>> >> getRegionLocation%28byte[],%20boolean%29
>> >>
>> >>
>> >> Š to know if you are continually hitting the same RS or spreading the
>> >>load.
>> >>
>> >>
>> >>
>> >> On 10/9/12 1:27 PM, "Mohit Anchlia" <[EMAIL PROTECTED]> wrote:
>> >>
>> >> >I just have 5 stress client threads writing timeseries data. What I
>> >>see is
>> >> >after few mts HBaseClient slows down and starts to take 4 secs. Once I
>> >> >kill
>> >> >the client and restart it stays at sustainable rate for about 2 mts
>> and
>> >> >then again it slows down. I am wondering if there is something I
>> >>should be
>> >> >doing on the HBaseclient side? All the request are similar in terms of
>> >> >data.
>> >>
>> >>
>> >>
>>
>>
>>
>

No methods for HTablePool to set 'auto flush'.
You can create a class to implement HTableInterfaceFactory. In method
'createHTableInterface'  table can be set auto flush.

2012/10/10 Mohit Anchlia <[EMAIL PROTECTED]>

> It looks as if RS is able to take the load but at some point memory buffer
> on the server is full and it slows everything down.
>
> Some interseting points I am seeing is memstore size of 50MB,
> fssynclatency_num_ops= 300k, fswritelatency=180k
>
> On Tue, Oct 9, 2012 at 11:03 AM, Mohit Anchlia <[EMAIL PROTECTED]
> >wrote:
>
> > There are 2 CF on 2 separate region server. And yes, I have not pre-split
> > the regions as I was told that we should let HBase handle that
> > automatically.
> >
> > Is there a way to set autoflush when using HTableDescriptor?
> >
> >  On Tue, Oct 9, 2012 at 10:50 AM, Doug Meil <
> [EMAIL PROTECTED]
> > > wrote:
> >
> >>
> >> So you're running on a single regionserver?
> >>
> >>
> >>
> >>
> >> On 10/9/12 1:44 PM, "Mohit Anchlia" <[EMAIL PROTECTED]> wrote:
> >>
> >> >I am using HTableInterface as a pool but I don't see any setautoflush
> >> >method. I am using 0.92.1 jar.
> >> >
> >> >Also, how can I see if RS is getting overloaded? I looked at the UI
> and I
> >> >don't see anything obvious:
> >> >
> >> >equestsPerSecond=0, numberOfOnlineRegions=1, numberOfStores=1,
> >> >numberOfStorefiles=1, storefileIndexSizeMB=0, rootIndexSizeKB=1,
> >> >totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, memstoreSizeMB=27,
> >> >readRequestsCount=126, writeRequestsCount=96157, compactionQueueSize=0,
> >> >flushQueueSize=0, usedHeapMB=44, maxHeapMB=3976, blockCacheSizeMB=8.79,
> >> >blockCacheFreeMB=985.34, blockCacheCount=11, blockCacheHitCount=23,
> >> >blockCacheMissCount=28, blockCacheEvictedCount=0,
> blockCacheHitRatio=45%,
> >> >blockCacheHitCachingRatio=67%, hdfsBlocksLocalityIndex=100
> >> >
> >> >On Tue, Oct 9, 2012 at 10:32 AM, Doug Meil
> >> ><[EMAIL PROTECTED]>wrote:
> >> >
> >> >>
> >> >> It's one of those "it depends" answers.
> >> >>
> >> >> See this firstŠ
> >> >>
> >> >> http://hbase.apache.org/book.html#perf.writing
> >> >>
> >> >> Š Additionally, one thing to understand is where you are writing
> data.
> >> >> Either keep track of the requests per RS over the period (e.g., the
> web
> >> >> interface), or you can also track it on the client side with...
> >> >>
> >> >>
> >> >>
> >>
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.htm
> >> >>l#
> >> >> getRegionLocation%28byte[],%20boolean%29
> >> >>
> >> >>
> >> >> Š to know if you are continually hitting the same RS or spreading the
> >> >>load.
> >> >>
> >> >>
> >> >>
> >> >> On 10/9/12 1:27 PM, "Mohit Anchlia" <[EMAIL PROTECTED]> wrote:
> >> >>
> >> >> >I just have 5 stress client threads writing timeseries data. What I
> >> >>see is
> >> >> >after few mts HBaseClient slows down and starts to take 4 secs.
> Once I
> >> >> >kill
> >> >> >the client and restart it stays at sustainable rate for about 2 mts
> >> and
> >> >> >then again it slows down. I am wondering if there is something I
> >> >>should be
> >> >> >doing on the HBaseclient side? All the request are similar in terms
> of
> >> >> >data.
> >> >>
> >> >>
> >> >>
> >>
> >>
> >>
> >
>