Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - reginoserver's log analysis?


+
hua beatls 2013-01-03, 09:45
+
Anoop Sam John 2013-01-03, 10:23
+
hua beatls 2013-01-03, 12:11
+
Anoop Sam John 2013-01-03, 12:37
+
hua beatls 2013-01-03, 16:21
+
ramkrishna vasudevan 2013-01-03, 16:24
+
Asaf Mesika 2013-01-03, 17:37
Copy link to this message
-
Re: reginoserver's log analysis?
hua beatls 2013-01-03, 18:25
HI,
   I think you have this experience, and i plan to increase the network
bandwith.
   but   some one told me that increase the
'hbase.hregion.memstore.block.multiplier" from default 2 to 4 could solve
this problem.
   which way to go?
   Thanks!
    bealts
On Fri, Jan 4, 2013 at 1:37 AM, Asaf Mesika <[EMAIL PROTECTED]> wrote:

> When I encountered this error it was caused by slow network which caused
> hdfs to be slow which caused flush to take more time thus blocking updates
> occurred.
>
> Sent from my iPhone
>
> On 3 בינו 2013, at 18:25, ramkrishna vasudevan <
> [EMAIL PROTECTED]> wrote:
>
> You need to see your application to see how much is your rate of puts.
> Also how many column families do you have?
> All these factors will help you in determining your memstore size.
>
> Regards
> Ram
>
> On Thu, Jan 3, 2013 at 9:51 PM, hua beatls <[EMAIL PROTECTED]> wrote:
>
> HI,
>
>   what i mean is shoud we increate the memstore according to this log?
>
>
>   Thanks!
>
>
>   beatls.
>
>
> On Thu, Jan 3, 2013 at 8:37 PM, Anoop Sam John <[EMAIL PROTECTED]> wrote:
>
>
>
> I guess you mean  "what is blocking memstore size"
>
> This you can configure using 2 properties.
>
> 1. hbase.hregion.memstore.flush.size -> Using this you specify when the
>
> memstore to be flushed as a file. The default value is 128MB. So when the
>
> region memstore size reaches this value a flush will be initiated.
>
> 2. hbase.hregion.memstore.block.multiplier -> deafult value is 2
>
> The blocking memstore size is calculated as flushSize multiplied by this
>
> multiplier.  So the default value will be 256MB.
>
> What value to set depends on the total memory available for the process
>
> in
>
> the RegionServers and the total no# of regions per server.
>
> Say you set this blocking size as 512m and there can be 100 regions per
>
> region server in your cluster, then you are *possibly allowing* a total
>
> memory of 512*100 mb for the memstores alone. [RS process will not other
>
> memory also for its normal operation].  Take care of these aspects when
>
> you
>
> design
>
>
> -Anoop-
>
>
> ________________________________________
>
> From: hua beatls [[EMAIL PROTECTED]]
>
> Sent: Thursday, January 03, 2013 5:41 PM
>
> To: [EMAIL PROTECTED]
>
> Subject: Re: reginoserver's log analysis?
>
>
> HI,
>
>    what is the block size? how to set it ?  and shoud i increate it for
>
> the memstore ?
>
>
>    Thanks!
>
>     beatls
>
>
> On Thu, Jan 3, 2013 at 6:23 PM, Anoop Sam John <[EMAIL PROTECTED]>
>
> wrote:
>
>
> The writes to your table will be written 1st to in memory data
>
> structure
>
> (memstore). When this memstore reaches some defined size it will be
>
> flushed
>
> into the file system and the memory can be cleared.
>
>
> The flush operation may take some time as IO write is involved, During
>
> this time also the client may be writing data to the table. Well HBase
>
> allows to do so and the data there in memory for a region can be
>
> becoming
>
> more and more. So there will be a max amount of total data which can be
>
> there in memory. If that is crossed HBase will stop taking any writes
>
> from
>
> the client until the flush is over and the memory got cleared.  As per
>
> the
>
> log in your cluster the blocking memstore size is 256m.
>
>
> Hope you got the idea.
>
>
> -Anoop-
>
> ________________________________________
>
> From: hua beatls [[EMAIL PROTECTED]]
>
> Sent: Thursday, January 03, 2013 3:15 PM
>
> To: [EMAIL PROTECTED]
>
> Subject: reginoserver's log analysis?
>
>
> Hi,
>
>   below is the regionserver's log, what does  the red marked "blocking
>
> updates" mean ?
>
>    any indication from " memstore size 256.8m is >= than blocking
>
> 256.0m
>
> size"
>
>
>
>
> 42,651 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking
>
> updatesfor 'IPC Server handler 1 on 60020' on region
>
>
>
>
>
> BT_NET_LOG_999,0000189020000000000000000000000000000000,1357201912899.efd6fd4581483aded26175300c48b7b7.:
>
> memstore size 256.8m is >= than blocking 256.0m size
+
Bryan Beaudreault 2013-01-03, 18:32
+
hua beatls 2013-01-03, 20:28