Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - metrics / performance impact


Copy link to this message
-
Re: metrics / performance impact
Nicolas Liochon 2012-11-26, 19:13
Hi Elliott,

Done: HBASE-7220

Turning the metrics off could be useful for testing. It's not very
difficult, because MetricsRegionWrapper is an interface, so we can have a
dummy implementation. I suppose all the other wrapper follow the same
pattern. I don't mind doing (well, it would be low priority task ;-) if the
idea itself is acceptable...

Thanks,

Nicolas
On Mon, Nov 26, 2012 at 7:46 PM, Elliott Clark <[EMAIL PROTECTED]> wrote:

> That's really weird.  All of those metrics were basically direct copies of
> what was there in 0.94 but using metrics2 to export things.  Can you file
> an issue and I'll try and take a look.  Last time I did a benchmark on the
> new metrics2 system I was getting a perf boost when compared to the
> metrics1 implementation.
>
> 1) There are still a few things to move over to metrics2 (rpc)
> 2) Completely turning metrics off is not possible currently.
>
>
> On Mon, Nov 26, 2012 at 10:21 AM, Nicolas Liochon <[EMAIL PROTECTED]>
> wrote:
>
> > Hi All,
> >
> > I've got an issue related to metrics in HBase trunk.
> > I'm trying to create a table with 3000 regions on two regions servers,
> from
> > the shell.
> >
> > It's ok on a standalone config.
> > It's ok on 0.94
> > It's not ok on trunk: it fails after around 1 hour.
> >
> > If I remove all the code related to metrics in HRegion, the 3000 regions
> > are created in 3 minutes (twice faster than the 0.94).
> >
> > When it fails the region server spends its time in "waitForWork", while
> the
> > master is in the tcp connection related code. It's a 1Gb network.
> >
> > I haven't looked at the metric code itself.
> >
> > So two questions :-)
> > 1) Is there any work in progress on the metrics? I agree that my case
> makes
> > no sense in production, but it's useful for testing (and may be there are
> > production uses cases that are impacted as well.)
> > 2) Is there any simple way to disable the metrics?
> >
> > See below the changes I made to HRegion
> >
> > Cheers,
> >
> > Nicolas
> >
> >
> > --
> >
> > index c70e9ab..6677e65 100644
> > ---
> >
> >
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> > +++
> >
> >
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> > @@ -364,7 +364,7 @@ public class HRegion implements HeapSize { // ,
> > Writable{
> >    private HTableDescriptor htableDescriptor = null;
> >    private RegionSplitPolicy splitPolicy;
> >
> > -  private final MetricsRegion metricsRegion;
> > +  private final MetricsRegion metricsRegion = null;
> >
> >    /**
> >     * Should only be used for testing purposes
> > @@ -388,7 +388,7 @@ public class HRegion implements HeapSize { // ,
> > Writable{
> >      this.coprocessorHost = null;
> >      this.scannerReadPoints = new ConcurrentHashMap<RegionScanner,
> Long>();
> >
> > -    this.metricsRegion = new MetricsRegion(new
> > MetricsRegionWrapperImpl(this));
> > +    //this.metricsRegion = new MetricsRegion(new
> > MetricsRegionWrapperImpl(this));
> >    }
> >
> >    /**
> > @@ -451,7 +451,7 @@ public class HRegion implements HeapSize { // ,
> > Writable{
> >      this.regiondir = getRegionDir(this.tableDir, encodedNameStr);
> >      this.scannerReadPoints = new ConcurrentHashMap<RegionScanner,
> Long>();
> >
> > -    this.metricsRegion = new MetricsRegion(new
> > MetricsRegionWrapperImpl(this));
> > +    //this.metricsRegion = new MetricsRegion(new
> > MetricsRegionWrapperImpl(this));
> >
> >      /*
> >       * timestamp.slop provides a server-side constraint on the
> timestamp.
> > This
> > @@ -1024,7 +1024,7 @@ public class HRegion implements HeapSize { // ,
> > Writable{
> >          status.setStatus("Running coprocessor post-close hooks");
> >          this.coprocessorHost.postClose(abort);
> >        }
> > -      this.metricsRegion.close();
> > +      //this.metricsRegion.close();
> >        status.markComplete("Closed");
> >        LOG.info("Closed " + this);
> >        return result;
> > @@ -2331,11 +2331,11 @@ public class HRegion implements HeapSize { // ,