Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Re: infinite loop of RS_ZK_REGION_SPLIT on .94.2


Copy link to this message
-
Re: infinite loop of RS_ZK_REGION_SPLIT on .94.2
Matt:
Was there any region server crash before you noticed the repeated log for
bc62a8a72124a4ba3f6b9f302587903c ?

I wonder if the following JIRA might be related:
HBASE-7083 SSH#fixupDaughter should force re-assign missing daughter

Thanks

On Sat, Nov 3, 2012 at 7:27 PM, Matt Corgan <[EMAIL PROTECTED]> wrote:

> Strangely, I don't see any record of that region in the master before what
> I already pasted even though I have logs back to 10/30.  Next time it
> happens I'll gather a full log record and try to debug while it's
> occurring.
>
>
> On Sat, Nov 3, 2012 at 7:10 PM, rajesh babu chintaguntla <
> [EMAIL PROTECTED]> wrote:
>
> > Hi Matt,
> > can you paste some more master logs of region
> > bc62a8a72124a4ba3f6b9f30258790 before split.
> > I think Its not problem with splitting.
> > We are getting
> >       LOG.warn("Region " + encodedName + " not found on server " +
> > serverName +
> >         "; failed processing");
> > this log means no entry in servers map(not fully assigned).
> >     Set<HRegionInfo> hris = this.servers.get(sn);
> >     HRegionInfo foundHri = null;
> >     for (HRegionInfo hri: hris) {
> >       if (hri.getEncodedName().equals(encodedName)) {
> >         foundHri = hri;
> >         break;
> >       }
> >     }
> >     return foundHri;
> >
> >
> >
> >
> > On Sun, Nov 4, 2012 at 6:07 AM, lars hofhansl <[EMAIL PROTECTED]>
> wrote:
> >
> > > CC'ing dev list...
> > >
> > > Is anybody aware of any changes that went in recently that could cause
> > > this?
> > > I looked around a bit, but could not find anything obvious.
> > >
> > > -- Lars
> > >
> > >
> > >
> > > ________________________________
> > >  From: Matt Corgan <[EMAIL PROTECTED]>
> > > To: user <[EMAIL PROTECTED]>
> > > Sent: Saturday, November 3, 2012 5:27 PM
> > > Subject: Re: infinite loop of RS_ZK_REGION_SPLIT on .94.2
> > >
> > > I think the cluster is ok without running hbck, as restarting the
> > > regionserver process stops the warnings and everything looks ok
> > otherwise.
> > >
> > > here's the regionserver right after the split happens:
> > > ------------------------
> > > 2012-11-01 22:45:28,726 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > > regionserver:60020-0x13ab46479832953 Attempting to transition node
> > > bc62a8a72124a4ba3f6b9f302587903c from *RS_ZK_R*
> > > *EGION_SPLITTING to RS_ZK_REGION_SPLIT*
> > > 2012-11-01 22:45:28,730 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > > regionserver:60020-0x13ab46479832953 Successfully transitioned node
> > > bc62a8a72124a4ba3f6b9f302587903c from RS_ZK_
> > > REGION_SPLITTING to RS_ZK_REGION_SPLIT
> > > 2012-11-01 22:45:28,730 DEBUG
> > > org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on
> > the
> > > master to process the split for bc62a8a72124a4ba3f6b9f302587903c
> > > 2012-11-01 22:45:28,832 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > > regionserver:60020-0x13ab46479832953 Attempting to transition node
> > > bc62a8a72124a4ba3f6b9f302587903c from RS_ZK_R
> > > EGION_SPLIT to RS_ZK_REGION_SPLIT
> > > 2012-11-01 22:45:28,837 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > > regionserver:60020-0x13ab46479832953 Successfully transitioned node
> > > bc62a8a72124a4ba3f6b9f302587903c from RS_ZK_
> > > REGION_SPLIT to RS_ZK_REGION_SPLIT
> > > -----------------------------
> > >
> > > The "transitioned node from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT"
> > > continues for 15 or so hours and finally settles without manual
> > > intervention with these regionserver log messages:
> > > -----------------------
> > > 2012-11-02 13:55:00,906 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > > regionserver:60020-0x13ab46479832953 Attempting to transition node *
> > > bc62a8a72124a4ba3f6b9f302587903c* from RS_ZK_REGION_SPLIT to
> > > RS_ZK_REGION_SPLIT
> > >
> > > 2012-11-02 13:55:00,916 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > > regionserver:60020-0x13ab46479832953 Successfully transitioned node *
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB