Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Regions failed to migrate


+
sagar naik 2011-12-05, 18:45
+
Jean-Daniel Cryans 2011-12-06, 18:01
Copy link to this message
-
Re: Regions failed to migrate
Shrijeet Paliwal 2011-12-06, 18:59
What J-D said or may be https://issues.apache.org/jira/browse/HBASE-4109 ?

On Tue, Dec 6, 2011 at 10:01 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> Your DNS is setup wrong:
>
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker: No HServerInfo
> found for datanode001,60020,1322010465849
>
> That's what that line means. The master did detect that a region
> server went down, but the name was unknown to it.
>
> J-D
>
> On Mon, Dec 5, 2011 at 10:45 AM, sagar naik <[EMAIL PROTECTED]> wrote:
> > Scenario:
> >  -  regionserver machine was rebooted . AWS random reboot .
> > Regionserver logs show shutdown
> >
> >  - Datanode which is on same machine also recvd a kill command
> >
> >  - Master did not migrate  the regions. It did detect the node down
> >
> >  - I checked after 6 hours , the hbase was in inconsistent state . (hbck)
> >
> >  - When I restarted the regionserver it was giving me a
> UnknownScannerException
> >
> >  - I ended up, restart master only and then regionserver would start fine
> >
> >  - After about 1 hr of regionserver going down, major compaction
> > (croned) kicked in
> >
> > Question:
> >
> >    Why did regions on the regionserver did not migrate ? am I missing
> > something, some config params.
> >    Most of the config is default except for compaction interval
> >
> > Thanks
> >
> >
> > REGIONSERVER LOGS:
> >
> > 2011-12-04 22:19:09,586 INFO
> > org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file
> >
> /user/nileus/hbase-storage/.logs/ip-X-X-X-X,60020,1322010465849/ip-10-174-43-151.us-west-1.compute.internal%3A60020.1322967102356
> > whose highest sequenceid is 127636589 to
> > /user/nileus/hbase-storage/.oldlogs/ip-X-X-x-X%3A60020.1322967102356
> > 2011-12-04 22:43:47,291 INFO
> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> > starting; hbase.shutdown.hook=true;
> > fsShutdownHook=Thread[Thread-15,5,main]
> > 2011-12-04 22:43:47,291 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
> > hook
> >
> >
> > MASTER LOG:
> > 2011-12-04 22:46:39,133 INFO
> > org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer
> > ephemeral node deleted, processing expiration
> > [datanode001,60020,1322010465849]
> > 2011-12-04 22:46:39,133 INFO
> > org.apache.hadoop.hbase.zookeeper.RegionServerTracker: No HServerInfo
> > found for datanode001,60020,1322010465849
> > 2011-12-04 22:47:20,828 INFO
> > org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
> > servers=3 regions=421 average=140.33333 mostloaded=141 leastloaded=141
> > 2011-12-04 22:52:20,893 INFO
> > org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
> > servers=3 regions=421 average=140.33333 mostloaded=141 leastloaded=141
> > 2011-12-04 22:57:20,959 INFO
> > org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
> > servers=3 regions=421 average=140.33333 mostloaded=141 leastloaded=141
> > 2
> >
> >
> > POST RESTART REGIONSERVER LOGS
> >
> > 2011-12-05 07:21:04,589 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > org.apache.hadoop.hbase.UnknownScannerException: Name: -1
> >        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1809)
> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >        at java.lang.reflect.Method.invoke(Method.java:597)
> >        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> >        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> > 2011-12-05 07:21:14,469 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > org.apache.hadoop.hbase.UnknownScannerException: Name: -1
> >        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1809)