Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - RPC statistics from region servers...


Copy link to this message
-
RE: RPC statistics from region servers...
Kannan Muthukkaruppan 2010-07-29, 23:17
I do seem to be getting data in multiPut* and get* now from all the Region Servers. So things look good now.

Gary:

> We could do a similar initialization of the JMX MBean by just introspecting
> the RPC interfaces though (HMasterInterface, HMasterRegionInterface,
> HRegionInterface).

 What would be the net effect of this change compared to the state trunk is currently in? Could you clarify?

Regards,
Kannan
-----Original Message-----
From: Gary Helmling [mailto:[EMAIL PROTECTED]]
Sent: Thursday, July 29, 2010 3:50 PM
To: [EMAIL PROTECTED]
Subject: Re: RPC statistics from region servers...

Kannan,

It looks like that code went away as part of HBASE-2219, which removed the
mapping of rpc method names to codes that was previously used in
serialization.

We could do a similar initialization of the JMX MBean by just introspecting
the RPC interfaces though (HMasterInterface, HMasterRegionInterface,
HRegionInterface).  Want to open a JIRA?

Gary
On Thu, Jul 29, 2010 at 3:16 PM, Kannan Muthukkaruppan
<[EMAIL PROTECTED]>wrote:

> Pushed the hadoop-metrics.properties changes, and it seems better but not
> quite. [It doesn't look like all the Region Servers are reporting similar
> set of counters (on some I am seeing "multiput" stuff, although we aren' t),
> lot of the average times are close to 0.]
>
> So next I looked into the HBASE-2146 patch, and that doesn't appear to be
> in trunk although at some point it was in trunk earlier.
>
>
> http://svn.apache.org/viewvc/hadoop/hbase/trunk/src/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java?p2=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&p1=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&r1=901402&r2=901401&view=diff&pathrev=901402
>
> Will try applying the patch locally to see if that helps.
>
> Regards,
> Kannan
> -----Original Message-----
> From: Kannan Muthukkaruppan [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, July 29, 2010 2:41 PM
> To: [EMAIL PROTECTED]
> Subject: RE: RPC statistics from region servers...
>
> Gary,
>
> With some recent restructuring of our conf/* and cluster deploy scripts, we
> did lose our hadoop-metrics.properties changes, and reverted back to the
> default one.  That could very well report the issue I ran into. I'll report
> back when I try it out.
>
> Thanks again for your help on this.
>
> Q: Why don't we make the suggested settings in
> http://hbase.apache.org/docs/current/metrics.html be part of the default
> conf/hadoop-metrics.properties?
>
> Regards,
> Kannan
> -----Original Message-----
> From: Gary Helmling [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, July 29, 2010 1:08 AM
> To: [EMAIL PROTECTED]
> Subject: Re: RPC statistics from region servers...
>
> Hmm, this sounds like HBASE-2146.  That patch had added code to initialize
> the RPC metrics MBean with a know list of RPC method names maintained in
> HBaseRPC.Invocation.  That way all per-method metrics would show up on
> server startup.  I believe that that method name mapping was later removed
> though (it can make the RPC serialization brittle, complicating things like
> rolling restarts), so maybe the patch that was applied to trunk was later
> removed?
>
> The MBean attributes for individual RPC methods _should_ be there after
> there has been activity involving those requests against the cluster.  If
> you start jconsole prior to the cluster activity however, I don't think the
> MBean definition will update, so you'll need to restart jconsole for them
> to
> show up.
>
> In addition, you should make sure you're running with a metrics context
> with
> an updater configured in
> conf/hadoop-metrics.properties (NullContextWithUpdateThread, FileContext,
> GangliaContext, etc).  And configure rpc.period with the update interval.
> These metrics don't show up as live counters in JMX, they're only pushed
> out
> to JMX during the periodic updates.
>
> If you've already configured all of those conditions and the metrics