St.Ack,

It's weird. I did Hbase upgrade from Hbase 0.20.3 to Hbase 0.90.2 by following "replace the Hadoop JAR file from Hadoop 0.20.2 release with the sync-supporting Hadoop JAR file found in HBase 0.90.2". It was suceesfully before, but now we are facing error from Hbase shell: NativeException: org.apache.hadoop.hbase.MasterNotRunningException: hofvmecln15.ecom.sears.com:60000, even though the processes of Hmaster and Region are running, and 60010 can be accessible via 60010.

Could you please help to find out the possible root cause? Below is the logging messages from the hbase.log from Master Node:

2011-07-14 22:54:55,903 INFO  zookeeper.ZooKeeper - Client environment:user.dir=/appl/hbase/share/hbase-0.90.2
2011-07-14 22:54:55,904 INFO  zookeeper.ZooKeeper - Initiating client connection, connectString=hofvmecln19:2181,hofvmecln18:2181,hofvmecln17:2181 sessionTimeout=60000 watcher=org.apache.hadoop.hbase.regionserver.HRegionServer@51b4a74b
2011-07-14 22:54:55,905 INFO  zookeeper.ClientCnxn - zookeeper.disableAutoWatchReset is false
2011-07-14 22:54:55,932 INFO  zookeeper.ClientCnxn - Attempting connection to server hofvmecln18/151.149.116.46:2181
2011-07-14 22:54:55,938 INFO  zookeeper.ClientCnxn - Priming connection to java.nio.channels.SocketChannel[connected local=/151.149.119.93:52156 remote=hofvmecln18/151.149.116.46:2181]
2011-07-14 22:54:55,939 INFO  zookeeper.ClientCnxn - Server connection successful
2011-07-14 22:54:56,030 INFO  regionserver.HRegionServer - Got ZooKeeper event, state: SyncConnected, type: None, path: null
2011-07-14 22:54:56,202 INFO  regionserver.HRegionServer - Telling master at hofvmecln15.ecom.sears.com:60000 that we are up
2011-07-14 22:54:56,407 WARN  ipc.HBaseServer - IPC Server listener on 60000: readAndProcess threw exception java.io.EOFException. Count of bytes read: 0
java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readUTF(DataInputStream.java:592)
        at java.io.DataInputStream.readUTF(DataInputStream.java:547)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Invocation.readFields(HBaseRPC.java:122)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:978)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:946)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
2011-07-14 22:54:56,412 WARN  regionserver.HRegionServer - Unable to connect to master. Retrying. Error was:
java.io.IOException: Call to hofvmecln15.ecom.sears.com/151.149.119.93:60000 failed on local exception: java.io.EOFException
        at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:757)
        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:727)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
        at $Proxy0.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:489)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:465)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:512)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:423)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1296)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1314)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:430)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:375)
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:504)
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:448)
thanks,
Andy Zhong

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Stack
Sent: Wednesday, June 08, 2011 11:53 AM
To: [EMAIL PROTECTED]
Subject: Re: Hadoop/HBase Upgrade Suggestion

On Wed, Jun 8, 2011 at 9:25 AM, Zhong, Sheng <[EMAIL PROTECTED]> wrote:

hadoop-0.20.203 does not have append support so its not recommended that you run hbase on it.  Here is our (a little stale at this point

St.Ack

This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.

Hey Andy,

This looks like the log from the regions server. Any chance you can post the
log for the HMaster?

-Joey

On Fri, Jul 15, 2011 at 12:43 AM, Zhong, Andy <[EMAIL PROTECTED]>wrote:

> St.Ack,
>
> It's weird. I did Hbase upgrade from Hbase 0.20.3 to Hbase 0.90.2 by
> following "replace the Hadoop JAR file from Hadoop 0.20.2 release with the
> sync-supporting Hadoop JAR file found in HBase 0.90.2". It was suceesfully
> before, but now we are facing error from Hbase shell: NativeException:
> org.apache.hadoop.hbase.MasterNotRunningException:
> hofvmecln15.ecom.sears.com:60000, even though the processes of Hmaster and
> Region are running, and 60010 can be accessible via 60010.
>
> Could you please help to find out the possible root cause? Below is the
> logging messages from the hbase.log from Master Node:
>
> 2011-07-14 22:54:55,903 INFO  zookeeper.ZooKeeper - Client
> environment:user.dir=/appl/hbase/share/hbase-0.90.2
> 2011-07-14 22:54:55,904 INFO  zookeeper.ZooKeeper - Initiating client
> connection, connectString=hofvmecln19:2181,hofvmecln18:2181,hofvmecln17:2181
> sessionTimeout=60000
> watcher=org.apache.hadoop.hbase.regionserver.HRegionServer@51b4a74b
> 2011-07-14 22:54:55,905 INFO  zookeeper.ClientCnxn -
> zookeeper.disableAutoWatchReset is false
> 2011-07-14 22:54:55,932 INFO  zookeeper.ClientCnxn - Attempting connection
> to server hofvmecln18/151.149.116.46:2181
> 2011-07-14 22:54:55,938 INFO  zookeeper.ClientCnxn - Priming connection to
> java.nio.channels.SocketChannel[connected local=/151.149.119.93:52156remote=hofvmecln18/
> 151.149.116.46:2181]
> 2011-07-14 22:54:55,939 INFO  zookeeper.ClientCnxn - Server connection
> successful
> 2011-07-14 22:54:56,030 INFO  regionserver.HRegionServer - Got ZooKeeper
> event, state: SyncConnected, type: None, path: null
> 2011-07-14 22:54:56,202 INFO  regionserver.HRegionServer - Telling master
> at hofvmecln15.ecom.sears.com:60000 that we are up
> 2011-07-14 22:54:56,407 WARN  ipc.HBaseServer - IPC Server listener on
> 60000: readAndProcess threw exception java.io.EOFException. Count of bytes
> read: 0
> java.io.EOFException
>        at java.io.DataInputStream.readFully(DataInputStream.java:180)
>        at java.io.DataInputStream.readUTF(DataInputStream.java:592)
>        at java.io.DataInputStream.readUTF(DataInputStream.java:547)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invocation.readFields(HBaseRPC.java:122)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:978)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:946)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:619)
> 2011-07-14 22:54:56,412 WARN  regionserver.HRegionServer - Unable to
> connect to master. Retrying. Error was:
> java.io.IOException: Call to
> hofvmecln15.ecom.sears.com/151.149.119.93:60000 failed on local exception:
> java.io.EOFException
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:757)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:727)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
>        at $Proxy0.getProtocolVersion(Unknown Source)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:489)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:465)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:512)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:423)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1296)

Joseph Echeverria
Cloudera, Inc.
443.305.9434