Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Balancer switch runs causing problems


Copy link to this message
-
Re: Balancer switch runs causing problems
/**
 * A janitor for the catalog tables.  Scans the <code>.META.</code> catalog
 * table on a period looking for unused regions to garbage collect.
 */
class CatalogJanitor extends Chore {
  private static final Log LOG =
LogFactory.getLog(CatalogJanitor.class.getName());
  private final Server server;
  private final MasterServices services;
  private boolean enabled = true;

  CatalogJanitor(final Server server, final MasterServices services) {
    super(server.getServerName() + "-CatalogJanitor",
      server.getConfiguration().getInt("hbase.catalogjanitor.interval",
300000),
....

Is it the above?

Enable RPC DEBUG-level logging on that server around the time that you see
the incidence and see if that helps give you a clue.

St.Ack

On Mon, Jan 27, 2014 at 8:56 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:

> But continue to see reads on META - no idea why ?
>
>
> On Mon, Jan 27, 2014 at 8:52 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:
>
> > We are not seeing any balancer related logs btw anymore...
> >
> >
> > On Mon, Jan 27, 2014 at 8:23 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> >> Looking at the changes since release 0.94.7, I found:
> >>
> >> HBASE-8655 Backport to 94 - HBASE-8346(Prefetching .META. rows in case
> >> only
> >> when useCache is set to true)
> >> HBASE-8698 potential thread creation in MetaScanner.metaScan
> >>
> >> If possible, can you upgrade your cluster ?
> >>
> >> Cheers
> >>
> >>
> >> On Mon, Jan 27, 2014 at 8:02 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >>
> >> > Do you see the following (from
> >> > HConnectionManager$HConnectionImplementation#locateRegionInMeta) ?
> >> >
> >> >             if (LOG.isDebugEnabled()) {
> >> >               LOG.debug("locateRegionInMeta parentTable=" +
> >> >                 Bytes.toString(parentTable) + ", metaLocation=" +
> >> >                 ((metaLocation == null)? "null": "{" + metaLocation +
> >> "}")
> >> > +
> >> >                 ", attempt=" + tries + " of " +
> >> >                 this.numRetries + " failed; retrying after sleep of "
> +
> >> >
> >> >
> >> > On Mon, Jan 27, 2014 at 7:51 PM, Varun Sharma <[EMAIL PROTECTED]>
> >> wrote:
> >> >
> >> >> Actually not sometimes but we are always seeing a large # of .META.
> >> reads
> >> >> every 5 minutes.
> >> >>
> >> >>
> >> >> On Mon, Jan 27, 2014 at 7:47 PM, Varun Sharma <[EMAIL PROTECTED]>
> >> >> wrote:
> >> >>
> >> >> > The default one with 0.94.7... - I dont see any of those logs. Also
> >> we
> >> >> > turned off the balancer switch - but looks like sometimes we still
> >> see a
> >> >> > large number of requests to .META. table every 5 minutes.
> >> >> >
> >> >> > Varun
> >> >> >
> >> >> >
> >> >> > On Mon, Jan 27, 2014 at 7:37 PM, Ted Yu <[EMAIL PROTECTED]>
> wrote:
> >> >> >
> >> >> >> In HMaster#balance(), we have (same for 0.94 and 0.96):
> >> >> >>
> >> >> >>         for (RegionPlan plan: plans) {
> >> >> >>           LOG.info("balance " + plan);
> >> >> >>
> >> >> >> Do you see such log in master log ?
> >> >> >>
> >> >> >>
> >> >> >> On Mon, Jan 27, 2014 at 7:26 PM, Varun Sharma <
> [EMAIL PROTECTED]>
> >> >> >> wrote:
> >> >> >>
> >> >> >> > We are seeing one other issue with high read latency (p99 etc.)
> on
> >> >> one
> >> >> >> of
> >> >> >> > our read heavy hbase clusters which is correlated with the
> >> balancer
> >> >> >> runs -
> >> >> >> > every 5 minutes.
> >> >> >> >
> >> >> >> > If there is no balancing to do, does the balancer only scan the
> >> table
> >> >> >> every
> >> >> >> > 5 minutes - does it do anything on top of that if the regions
> are
> >> >> >> balanced
> >> >> >> > ?
> >> >> >> >
> >> >> >> > Varun
> >> >> >> >
> >> >> >>
> >> >> >
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB