Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - Re: infinite loop of RS_ZK_REGION_SPLIT on .94.2


Copy link to this message
-
Re: infinite loop of RS_ZK_REGION_SPLIT on .94.2
ramkrishna vasudevan 2012-11-04, 17:12
Seems strange.  I too will look into this.  Backtracking the region and its
parent will give an idea.

Regards
Ram

On Sun, Nov 4, 2012 at 7:57 AM, Matt Corgan <[EMAIL PROTECTED]> wrote:

> Strangely, I don't see any record of that region in the master before what
> I already pasted even though I have logs back to 10/30.  Next time it
> happens I'll gather a full log record and try to debug while it's
> occurring.
>
>
> On Sat, Nov 3, 2012 at 7:10 PM, rajesh babu chintaguntla <
> [EMAIL PROTECTED]> wrote:
>
> > Hi Matt,
> > can you paste some more master logs of region
> > bc62a8a72124a4ba3f6b9f30258790 before split.
> > I think Its not problem with splitting.
> > We are getting
> >       LOG.warn("Region " + encodedName + " not found on server " +
> > serverName +
> >         "; failed processing");
> > this log means no entry in servers map(not fully assigned).
> >     Set<HRegionInfo> hris = this.servers.get(sn);
> >     HRegionInfo foundHri = null;
> >     for (HRegionInfo hri: hris) {
> >       if (hri.getEncodedName().equals(encodedName)) {
> >         foundHri = hri;
> >         break;
> >       }
> >     }
> >     return foundHri;
> >
> >
> >
> >
> > On Sun, Nov 4, 2012 at 6:07 AM, lars hofhansl <[EMAIL PROTECTED]>
> wrote:
> >
> > > CC'ing dev list...
> > >
> > > Is anybody aware of any changes that went in recently that could cause
> > > this?
> > > I looked around a bit, but could not find anything obvious.
> > >
> > > -- Lars
> > >
> > >
> > >
> > > ________________________________
> > >  From: Matt Corgan <[EMAIL PROTECTED]>
> > > To: user <[EMAIL PROTECTED]>
> > > Sent: Saturday, November 3, 2012 5:27 PM
> > > Subject: Re: infinite loop of RS_ZK_REGION_SPLIT on .94.2
> > >
> > > I think the cluster is ok without running hbck, as restarting the
> > > regionserver process stops the warnings and everything looks ok
> > otherwise.
> > >
> > > here's the regionserver right after the split happens:
> > > ------------------------
> > > 2012-11-01 22:45:28,726 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > > regionserver:60020-0x13ab46479832953 Attempting to transition node
> > > bc62a8a72124a4ba3f6b9f302587903c from *RS_ZK_R*
> > > *EGION_SPLITTING to RS_ZK_REGION_SPLIT*
> > > 2012-11-01 22:45:28,730 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > > regionserver:60020-0x13ab46479832953 Successfully transitioned node
> > > bc62a8a72124a4ba3f6b9f302587903c from RS_ZK_
> > > REGION_SPLITTING to RS_ZK_REGION_SPLIT
> > > 2012-11-01 22:45:28,730 DEBUG
> > > org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on
> > the
> > > master to process the split for bc62a8a72124a4ba3f6b9f302587903c
> > > 2012-11-01 22:45:28,832 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > > regionserver:60020-0x13ab46479832953 Attempting to transition node
> > > bc62a8a72124a4ba3f6b9f302587903c from RS_ZK_R
> > > EGION_SPLIT to RS_ZK_REGION_SPLIT
> > > 2012-11-01 22:45:28,837 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > > regionserver:60020-0x13ab46479832953 Successfully transitioned node
> > > bc62a8a72124a4ba3f6b9f302587903c from RS_ZK_
> > > REGION_SPLIT to RS_ZK_REGION_SPLIT
> > > -----------------------------
> > >
> > > The "transitioned node from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT"
> > > continues for 15 or so hours and finally settles without manual
> > > intervention with these regionserver log messages:
> > > -----------------------
> > > 2012-11-02 13:55:00,906 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > > regionserver:60020-0x13ab46479832953 Attempting to transition node *
> > > bc62a8a72124a4ba3f6b9f302587903c* from RS_ZK_REGION_SPLIT to
> > > RS_ZK_REGION_SPLIT
> > >
> > > 2012-11-02 13:55:00,916 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > > regionserver:60020-0x13ab46479832953 Successfully transitioned node *
> > > bc62a8a72124a4ba3f6b9f302587903c* from RS_ZK_REGION_SPLIT to
> > > RS_ZK_REGION_SPLIT
> > >
> > > 2012-11-02 13:55:00,916 INFO