Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - reginoserver's log analysis?


+
hua beatls 2013-01-03, 09:45
+
Anoop Sam John 2013-01-03, 10:23
+
hua beatls 2013-01-03, 12:11
+
Anoop Sam John 2013-01-03, 12:37
Copy link to this message
-
Re: reginoserver's log analysis?
hua beatls 2013-01-03, 16:21
HI,
   what i mean is shoud we increate the memstore according to this log?

   Thanks!

   beatls.

On Thu, Jan 3, 2013 at 8:37 PM, Anoop Sam John <[EMAIL PROTECTED]> wrote:

>
> I guess you mean  "what is blocking memstore size"
> This you can configure using 2 properties.
> 1. hbase.hregion.memstore.flush.size -> Using this you specify when the
> memstore to be flushed as a file. The default value is 128MB. So when the
> region memstore size reaches this value a flush will be initiated.
> 2. hbase.hregion.memstore.block.multiplier -> deafult value is 2
> The blocking memstore size is calculated as flushSize multiplied by this
>  multiplier.  So the default value will be 256MB.
> What value to set depends on the total memory available for the process in
> the RegionServers and the total no# of regions per server.
> Say you set this blocking size as 512m and there can be 100 regions per
> region server in your cluster, then you are *possibly allowing* a total
> memory of 512*100 mb for the memstores alone. [RS process will not other
> memory also for its normal operation].  Take care of these aspects when you
> design
>
> -Anoop-
>
> ________________________________________
> From: hua beatls [[EMAIL PROTECTED]]
> Sent: Thursday, January 03, 2013 5:41 PM
> To: [EMAIL PROTECTED]
> Subject: Re: reginoserver's log analysis?
>
> HI,
>     what is the block size? how to set it ?  and shoud i increate it for
> the memstore ?
>
>     Thanks!
>      beatls
>
> On Thu, Jan 3, 2013 at 6:23 PM, Anoop Sam John <[EMAIL PROTECTED]> wrote:
>
> > The writes to your table will be written 1st to in memory data structure
> > (memstore). When this memstore reaches some defined size it will be
> flushed
> > into the file system and the memory can be cleared.
> >
> > The flush operation may take some time as IO write is involved, During
> > this time also the client may be writing data to the table. Well HBase
> > allows to do so and the data there in memory for a region can be becoming
> > more and more. So there will be a max amount of total data which can be
> > there in memory. If that is crossed HBase will stop taking any writes
> from
> > the client until the flush is over and the memory got cleared.  As per
> the
> > log in your cluster the blocking memstore size is 256m.
> >
> > Hope you got the idea.
> >
> > -Anoop-
> > ________________________________________
> > From: hua beatls [[EMAIL PROTECTED]]
> > Sent: Thursday, January 03, 2013 3:15 PM
> > To: [EMAIL PROTECTED]
> > Subject: reginoserver's log analysis?
> >
> > Hi,
> >    below is the regionserver's log, what does  the red marked "blocking
> > updates" mean ?
> >     any indication from " memstore size 256.8m is >= than blocking 256.0m
> > size"
> >
> >
> >
> > 42,651 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking
> > updatesfor 'IPC Server handler 1 on 60020' on region
> >
> >
> BT_NET_LOG_999,0000189020000000000000000000000000000000,1357201912899.efd6fd4581483aded26175300c48b7b7.:
> > memstore size 256.8m is >= than blocking 256.0m size
> >
> > 2013-01-03 17:00:42,717 INFO
> > org.apache.hadoop.hbase.regionserver.HRegion: Blocking
> > updates for 'IPC Server handler 27 on 60020' on region
> >
> >
> BT_NET_LOG_999,0000189020000000000000000000000000000000,1357201912899.efd6fd4581483aded26175300c48b7b7.:
> > memstore size 256.8m is >= than blocking 256.0m size
> >
> > 2013-01-03 17:00:42,761 INFO
> org.apache.hadoop.hbase.regionserver.HRegion:
> > Blocking updates for 'IPC Server handler 50 on 60020' on region
> >
> >
> BT_NET_LOG_999,0000189020000000000000000000000000000000,1357201912899.efd6fd4581483aded26175300c48b7b7.:
> > memstore size 256.8m is >= than blocking 256.0m size
> >
> > 2013-01-03 17:00:42,875 INFO
> org.apache.hadoop.hbase.regionserver.HRegion:
> > Blocking updates for 'IPC Server handler 55 on 60020' on region
> >
> >
> BT_NET_LOG_999,0000189020000000000000000000000000000000,1357201912899.efd6fd4581483aded26175300c48b7b7.:
> > memstore size 256.8m is >= than blocking 256.0m size
+
ramkrishna vasudevan 2013-01-03, 16:24
+
Asaf Mesika 2013-01-03, 17:37
+
hua beatls 2013-01-03, 18:25
+
Bryan Beaudreault 2013-01-03, 18:32
+
hua beatls 2013-01-03, 20:28