Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> AssignmentManager looping?


+
Jean-Marc Spaggiari 2013-08-01, 12:22
+
Kevin Odell 2013-08-01, 12:42
+
Jean-Marc Spaggiari 2013-08-01, 12:51
+
Kevin Odell 2013-08-01, 13:25
+
Jean-Marc Spaggiari 2013-08-01, 14:08
Copy link to this message
-
Re: AssignmentManager looping?
Something went wrong with split.  It should be easy to fix your cluster.
However, it will be more interesting to find out how it happened. Do you
remember what has happened since it was good previously? Do you have all
the logs?
On Thu, Aug 1, 2013 at 7:08 AM, Jean-Marc Spaggiari <[EMAIL PROTECTED]
> wrote:

> I tried to remove the znodes but got the same result. So I shutted down all
> the RS and restarted HBase, and now I have 0 regions for this table.
> Running HBCK. Seems that it has a lot to do...
>
> 2013/8/1 Kevin O'dell <[EMAIL PROTECTED]>
>
> > Yes you can if HBase is down, first I would copy .META out of HDFS local
> > and then you can search it for split issues. Deleting those znodes should
> > clear this up though.
> > On Aug 1, 2013 8:52 AM, "Jean-Marc Spaggiari" <[EMAIL PROTECTED]>
> > wrote:
> >
> > > I can't check the meta since HBase is down.
> > >
> > > Regarding HDFS, I took few random lines like:
> > > 2013-08-01 08:45:57,260 WARN
> > > org.apache.hadoop.hbase.master.AssignmentManager: Region
> > > 28328fdb7181cbd9cc4d6814775e8895 not found on server
> > > node4,60020,1375319042033; failed processing
> > > 2013-08-01 08:45:57,260 WARN
> > > org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for
> > region
> > > 28328fdb7181cbd9cc4d6814775e8895 from server node4,60020,1375319042033
> > but
> > > it doesn't exist anymore, probably already processed its split
> > >
> > > And each time, there is nothing like that.
> > > hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -lsr / | grep
> > > 28328fdb7181cbd9cc4d6814775e8895
> > >
> > > On ZK side:
> > > [zk: localhost:2181(CONNECTED) 3] ls /hbase/splitlog
> > >
> > > [zk: localhost:2181(CONNECTED) 10] ls /hbase/unassigned
> > > [28328fdb7181cbd9cc4d6814775e8895, a8781a598c46f19723a2405345b58470,
> > > b7ebfeb63b10997736fd12920fde2bb8, d95bb27cc026511c2a8c8ad155e79bf6,
> > > 270a9c371fcbe9cd9a04986e0b77d16b, aff4d1d8bf470458bb19525e8aef0759]
> > >
> > > Can I just delete those zknodes? Worst case hbck will find them back
> from
> > > HDFS if required?
> > >
> > > JM
> > >
> > > 2013/8/1 Kevin O'dell <[EMAIL PROTECTED]>
> > >
> > > > Does it exist in meta or hdfs?
> > > > On Aug 1, 2013 8:24 AM, "Jean-Marc Spaggiari" <
> [EMAIL PROTECTED]
> > >
> > > > wrote:
> > > >
> > > > > My master keep logging that:
> > > > >
> > > > > 2013-07-31 21:52:59,201 WARN
> > > > > org.apache.hadoop.hbase.master.AssignmentManager: Region
> > > > > 270a9c371fcbe9cd9a04986e0b77d16b not found on server
> > > > > node7,60020,1375319044055; failed processing
> > > > > 2013-07-31 21:52:59,201 WARN
> > > > > org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT
> for
> > > > region
> > > > > 270a9c371fcbe9cd9a04986e0b77d16b from server
> > node7,60020,1375319044055
> > > > but
> > > > > it doesn't exist anymore, probably already processed its split
> > > > > 2013-07-31 21:52:59,339 WARN
> > > > > org.apache.hadoop.hbase.master.AssignmentManager: Region
> > > > > 270a9c371fcbe9cd9a04986e0b77d16b not found on server
> > > > > node7,60020,1375319044055; failed processing
> > > > > 2013-07-31 21:52:59,339 WARN
> > > > > org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT
> for
> > > > region
> > > > > 270a9c371fcbe9cd9a04986e0b77d16b from server
> > node7,60020,1375319044055
> > > > but
> > > > > it doesn't exist anymore, probably already processed its split
> > > > > 2013-07-31 21:52:59,461 WARN
> > > > > org.apache.hadoop.hbase.master.AssignmentManager: Region
> > > > > 270a9c371fcbe9cd9a04986e0b77d16b not found on server
> > > > > node7,60020,1375319044055; failed processing
> > > > > 2013-07-31 21:52:59,461 WARN
> > > > > org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT
> for
> > > > region
> > > > > 270a9c371fcbe9cd9a04986e0b77d16b from server
> > node7,60020,1375319044055
> > > > but
> > > > > it doesn't exist anymore, probably already processed its split
> > > > > 2013-07-31 21:52:59,636 WARN
> > > > > org.apache.hadoop.hbase.master.AssignmentManager: Region
+
Jean-Marc Spaggiari 2013-08-01, 17:00
+
Kevin Odell 2013-08-01, 17:38
+
Kevin Odell 2013-08-01, 17:39
+
Jean-Marc Spaggiari 2013-08-01, 17:48
+
Kevin Odell 2013-08-01, 17:52
+
Jean-Marc Spaggiari 2013-08-01, 18:07
+
Jimmy Xiang 2013-08-01, 18:29