Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> RPC statistics from region servers...


Copy link to this message
-
RE: RPC statistics from region servers...
I do seem to be getting data in multiPut* and get* now from all the Region Servers. So things look good now.

Gary:

> We could do a similar initialization of the JMX MBean by just introspecting
> the RPC interfaces though (HMasterInterface, HMasterRegionInterface,
> HRegionInterface).

 What would be the net effect of this change compared to the state trunk is currently in? Could you clarify?

Regards,
Kannan
-----Original Message-----
From: Gary Helmling [mailto:[EMAIL PROTECTED]]
Sent: Thursday, July 29, 2010 3:50 PM
To: [EMAIL PROTECTED]
Subject: Re: RPC statistics from region servers...

Kannan,

It looks like that code went away as part of HBASE-2219, which removed the
mapping of rpc method names to codes that was previously used in
serialization.

We could do a similar initialization of the JMX MBean by just introspecting
the RPC interfaces though (HMasterInterface, HMasterRegionInterface,
HRegionInterface).  Want to open a JIRA?

Gary
On Thu, Jul 29, 2010 at 3:16 PM, Kannan Muthukkaruppan
<[EMAIL PROTECTED]>wrote:

> Pushed the hadoop-metrics.properties changes, and it seems better but not
> quite. [It doesn't look like all the Region Servers are reporting similar
> set of counters (on some I am seeing "multiput" stuff, although we aren' t),
> lot of the average times are close to 0.]
>
> So next I looked into the HBASE-2146 patch, and that doesn't appear to be
> in trunk although at some point it was in trunk earlier.
>
>
> http://svn.apache.org/viewvc/hadoop/hbase/trunk/src/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java?p2=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&p1=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&r1=901402&r2=901401&view=diff&pathrev=901402
>
> Will try applying the patch locally to see if that helps.
>
> Regards,
> Kannan
> -----Original Message-----
> From: Kannan Muthukkaruppan [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, July 29, 2010 2:41 PM
> To: [EMAIL PROTECTED]
> Subject: RE: RPC statistics from region servers...
>
> Gary,
>
> With some recent restructuring of our conf/* and cluster deploy scripts, we
> did lose our hadoop-metrics.properties changes, and reverted back to the
> default one.  That could very well report the issue I ran into. I'll report
> back when I try it out.
>
> Thanks again for your help on this.
>
> Q: Why don't we make the suggested settings in
> http://hbase.apache.org/docs/current/metrics.html be part of the default
> conf/hadoop-metrics.properties?
>
> Regards,
> Kannan
> -----Original Message-----
> From: Gary Helmling [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, July 29, 2010 1:08 AM
> To: [EMAIL PROTECTED]
> Subject: Re: RPC statistics from region servers...
>
> Hmm, this sounds like HBASE-2146.  That patch had added code to initialize
> the RPC metrics MBean with a know list of RPC method names maintained in
> HBaseRPC.Invocation.  That way all per-method metrics would show up on
> server startup.  I believe that that method name mapping was later removed
> though (it can make the RPC serialization brittle, complicating things like
> rolling restarts), so maybe the patch that was applied to trunk was later
> removed?
>
> The MBean attributes for individual RPC methods _should_ be there after
> there has been activity involving those requests against the cluster.  If
> you start jconsole prior to the cluster activity however, I don't think the
> MBean definition will update, so you'll need to restart jconsole for them
> to
> show up.
>
> In addition, you should make sure you're running with a metrics context
> with
> an updater configured in
> conf/hadoop-metrics.properties (NullContextWithUpdateThread, FileContext,
> GangliaContext, etc).  And configure rpc.period with the update interval.
> These metrics don't show up as live counters in JMX, they're only pushed
> out
> to JMX during the periodic updates.
>
> If you've already configured all of those conditions and the metrics
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB