Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - Re: infinite loop of RS_ZK_REGION_SPLIT on .94.2


Copy link to this message
-
Re: infinite loop of RS_ZK_REGION_SPLIT on .94.2
Matt Corgan 2012-11-04, 02:27
Strangely, I don't see any record of that region in the master before what
I already pasted even though I have logs back to 10/30.  Next time it
happens I'll gather a full log record and try to debug while it's occurring.
On Sat, Nov 3, 2012 at 7:10 PM, rajesh babu chintaguntla <
[EMAIL PROTECTED]> wrote:

> Hi Matt,
> can you paste some more master logs of region
> bc62a8a72124a4ba3f6b9f30258790 before split.
> I think Its not problem with splitting.
> We are getting
>       LOG.warn("Region " + encodedName + " not found on server " +
> serverName +
>         "; failed processing");
> this log means no entry in servers map(not fully assigned).
>     Set<HRegionInfo> hris = this.servers.get(sn);
>     HRegionInfo foundHri = null;
>     for (HRegionInfo hri: hris) {
>       if (hri.getEncodedName().equals(encodedName)) {
>         foundHri = hri;
>         break;
>       }
>     }
>     return foundHri;
>
>
>
>
> On Sun, Nov 4, 2012 at 6:07 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>
> > CC'ing dev list...
> >
> > Is anybody aware of any changes that went in recently that could cause
> > this?
> > I looked around a bit, but could not find anything obvious.
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> >  From: Matt Corgan <[EMAIL PROTECTED]>
> > To: user <[EMAIL PROTECTED]>
> > Sent: Saturday, November 3, 2012 5:27 PM
> > Subject: Re: infinite loop of RS_ZK_REGION_SPLIT on .94.2
> >
> > I think the cluster is ok without running hbck, as restarting the
> > regionserver process stops the warnings and everything looks ok
> otherwise.
> >
> > here's the regionserver right after the split happens:
> > ------------------------
> > 2012-11-01 22:45:28,726 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > regionserver:60020-0x13ab46479832953 Attempting to transition node
> > bc62a8a72124a4ba3f6b9f302587903c from *RS_ZK_R*
> > *EGION_SPLITTING to RS_ZK_REGION_SPLIT*
> > 2012-11-01 22:45:28,730 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > regionserver:60020-0x13ab46479832953 Successfully transitioned node
> > bc62a8a72124a4ba3f6b9f302587903c from RS_ZK_
> > REGION_SPLITTING to RS_ZK_REGION_SPLIT
> > 2012-11-01 22:45:28,730 DEBUG
> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on
> the
> > master to process the split for bc62a8a72124a4ba3f6b9f302587903c
> > 2012-11-01 22:45:28,832 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > regionserver:60020-0x13ab46479832953 Attempting to transition node
> > bc62a8a72124a4ba3f6b9f302587903c from RS_ZK_R
> > EGION_SPLIT to RS_ZK_REGION_SPLIT
> > 2012-11-01 22:45:28,837 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > regionserver:60020-0x13ab46479832953 Successfully transitioned node
> > bc62a8a72124a4ba3f6b9f302587903c from RS_ZK_
> > REGION_SPLIT to RS_ZK_REGION_SPLIT
> > -----------------------------
> >
> > The "transitioned node from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT"
> > continues for 15 or so hours and finally settles without manual
> > intervention with these regionserver log messages:
> > -----------------------
> > 2012-11-02 13:55:00,906 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > regionserver:60020-0x13ab46479832953 Attempting to transition node *
> > bc62a8a72124a4ba3f6b9f302587903c* from RS_ZK_REGION_SPLIT to
> > RS_ZK_REGION_SPLIT
> >
> > 2012-11-02 13:55:00,916 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > regionserver:60020-0x13ab46479832953 Successfully transitioned node *
> > bc62a8a72124a4ba3f6b9f302587903c* from RS_ZK_REGION_SPLIT to
> > RS_ZK_REGION_SPLIT
> >
> > 2012-11-02 13:55:00,916 INFO
> > org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
> > updated, and report to master.
> >
> >
> Parent=ActiveListingRecord16,\x83\x07\xDC\x07\x01Obeo\x00690461,1351816858693.
> > *bc62a8a72124a4ba3f6b9f302587903c*., new regions:
> >
> >
> ActiveListingRecord16,\x83\x07\xDC\x07\x01Obeo\x00690461,1351824327023.22c3fa48d17aa7312ca53566c680f0fd.,
> >
> >
> ActiveListingRecord16,\x83\x07\xDC\x07\x11WebsiteIDX\x009024215,1351824327023.b0e0a488c711e5c7f74ee6198a9755a2..