Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> RPC - Queue Time when handlers are all waiting


Copy link to this message
-
Re: RPC - Queue Time when handlers are all waiting
I thought we could, somehow, check queue items. Replication Handlers status
are waiting most of the time:
Here's a sample of 1 minute, refreshing every second. Allways got same
output while RpcQueueTime_avg_time was reporting activity (Handlers 29...0
were all WAITING).:

hbase@pam-hb-replica-b-00:~$ jstack 15585 | grep 'IPC Server handler' -A 20
| egrep 'RUNNABLE|BLOCKED' -B 1 -A 20
"IPC Server listener on 60020" daemon prio=10 tid=0x00007f3658189000
nid=0x3d3a runnable [0x00007f363ca68000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000070030f3a8> (a sun.nio.ch.Util$2)
- locked <0x000000070030f390> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007003d31b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:636)

Thanks!

2013/12/10 Jean-Marc Spaggiari <[EMAIL PROTECTED]>

> You can do something like (types it in the email, did not tested it):
>
> for i in {1..240}; do wget "
> http://MASTER_IP:6001/master-status?format=json&filter=handler"; done
>
> That will create a 240 JSon files that you can look at to see handlers
> status. Also, you this same page, you should see you 30 replication
> handlers as well as you 30 "standard" handlers.
>
> Last, I will recommend you to move back to less replication handlers since
> I don't think you need 30.
>
> JM
>
>
> 2013/12/10 Federico Gaule <[EMAIL PROTECTED]>
>
> > There is any coprocessor on slaves, neither master.
> > How can i dump RPC queues?
> >
> > Thanks!
> >
> >
> > 2013/12/10 Jean-Marc Spaggiari <[EMAIL PROTECTED]>
> >
> > > Here are the properties from the code regarding the handlers.
> > > hbase.master.handler.count
> > > hbase.regionserver.handler.count
> > > hbase.regionserver.replication.handler.count
> > > hbase.regionserver.metahandler.count
> > >
> > > Do you have any coprocessor configured on your slave cluster? Ar eyou
> > able
> > > to dump the RPC queues every 5 seconds to see what is in?
> > >
> > > JM
> > >
> > >
> > >
> > > 2013/12/10 Federico Gaule <[EMAIL PROTECTED]>
> > >
> > > > I'm using hbase 0.94.13.
> > > > hbase.regionserver.metahandler.count is a more intuituve name for
> those
> > > > handlers :)
> > > >
> > > >
> > > > 2013/12/10 Nicolas Liochon <[EMAIL PROTECTED]>
> > > >
> > > > > It's hbase.regionserver.metahandler.count. Not sure it causes the
> > issue
> > > > > you're facing, thought. What's your HBase version?
> > > > >
> > > > >
> > > > > On Tue, Dec 10, 2013 at 1:21 PM, Federico Gaule <
> [EMAIL PROTECTED]
> > >
> > > > > wrote:
> > > > >
> > > > > > There is another set of handler we haven't customized "PRI IPC"
> > > > (priority
> > > > > > ?). What are those handlers used for? What is the property used
> to
> > > > > increase
> > > > > > the number of handlers?
> > > > hbase.regionserver.custom.priority.handler.count
> > > > > ?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > >
> > > > > > 2013/12/10 Federico Gaule <[EMAIL PROTECTED]>
> > > > > >
> > > > > > > I've increased hbase.regionserver.replication.handler.count 10x
> > > (30)
> > > > > but
> > > > > > > nothing have changed. rpc.metrics.RpcQueueTime_avg_time still
> > shows
> > > > > > > activity :(
> > > > > > >
> > > > > > > Mon Dec 09 14:04:10 EST 2013REPL IPC Server handler 29 on 60000
> > > > WAITING
> > > > > > > (since 16hrs, 58mins, 56sec ago)Waiting for a call (since
> 16hrs,
> > > > > 58mins,
> > > > > > > 56sec ago) Mon Dec 09 14:04:10 EST 2013 REPL IPC Server handler
> > 28
> > > on
> > > > > > > 60000WAITING (since 16hrs, 58mins, 56sec ago) Waiting for a
> call
> > > > (since
> > > > > > > 16hrs, 58mins, 56sec ago)Mon Dec 09 14:04:10 EST 2013 REPL IPC
[image: http://www.despegar.com/galeria/images/promos/isodespegar1.png]

*Ing. Federico Gaule*
Líder Técnico - PAM <[EMAIL PROTECTED]>
Av. Corrientes 746 - Piso 9 - C.A.B.A. (C1043AAU)
tel. +54 (11) 4894-3500

*[image: Seguinos en Twitter!] <http://twitter.com/#!/despegarar> [image:
Seguinos en Facebook!] <http://www.facebook.com/despegar> [image: Seguinos
en YouTube!] <http://www.youtube.com/Despegar>*
*Despegar.com, Inc. *
El mejor precio para tu viaje.

Este mensaje es confidencial y puede contener información amparada por el
secreto profesional.
Si usted ha recibido este e-mail por error, por favor comuníquenoslo
inmediatamente respondiendo a este e-mail y luego eliminándolo de su
sistema.
El contenido de este mensaje no deberá ser copiado ni divulgado a ninguna
persona.