Hi
I try to use hbase, and now ,I have a problem with hbase
configuration, I use 8node for try, and it seems to work, hadoop, zookeeper,
hbase all boot up, and it could insert into with putAPI ,
But when I try to use masterIP:60010 to manage it, I find there are only one
regionserver there, and it is localhost:60030, why?
I have set regionserver file , there are 8 nodes there, and all
could ssh to each other without passwd,

And I also use hdfsNamenodeIP:50070 to see HDFS, and it seems OK,
the data have been balanced across 8node, but I wander hbase only have one
regionserver to work, although when I star/stop regionserver , there are
8region server to start and stop,

And I try to put data into hbase, it seems ok at first, but after
200million in hbase, it seems really hard to insert more into it, it is very
slow, and I use masterIP:60010 to manage it, I find there are only one
regionserver there, and it is localhost:60030, why?


Could you help me,
 


Thank you
-------------------
Tian Guanhua

For starters, you'll need to sync up the clocks on all your machines.
Install ntp or similar and those ClockOutOfSync exceptions will clear up.
Specifically, blade4 looks off.

-n
On Dec 10, 2012 6:18 PM, "tgh" <[EMAIL PROTECTED]> wrote:

> Meanwhile , log is master , that is, blade1, is like this,  there are some
> ERRor like this, for
>
> 2012-09-01 06:31:05,558 INFO org.apache.hadoop.hbase.master.ServerManager:
> Registering server=blade2,60020,1346452716636, regionCount=0, userLoad=false
> 2012-09-01 06:31:05,569 WARN org.apache.hadoop.hbase.master.ServerManager:
> Server blade4,60020,1346451768443 has been rejected; Reported time is too
> far out of sync with master.  Time difference of 496371ms > max allowed of
> 30000ms
> 2012-09-01 06:31:05,581 WARN org.apache.hadoop.hbase.master.ServerManager:
> Server blade5,60020,1346452001672 has been rejected; Reported time is too
> far out of sync with master.  Time difference of 263137ms > max allowed of
> 30000ms
> 2012-09-01 06:31:05,583 ERROR org.apache.hadoop.hbase.master.HMaster:
> Region server serverName=blade4,60020,1346451768443, load=(requests=0,
> regions=0, usedHeap=142, maxHeap=966) reported a fatal error:
> ABORTING region server serverName=blade4,60020,1346451768443,
> load=(requests=0, regions=0, usedHeap=142, maxHeap=966): Unhandled
> exception: org.apache.hadoop.hbase.ClockOutOfSyncException: Server
> blade4,60020,1346451768443 has been rejected; Reported time is too far out
> of sync with master.  Time difference of 496371ms > max allowed of 30000ms
> Cause:
> org.apache.hadoop.hbase.ClockOutOfSyncException:
> org.apache.hadoop.hbase.ClockOutOfSyncException: Server
> blade4,60020,1346451768443 has been rejected; Reported time is too far out
> of sync with master.  Time difference of 496371ms > max allowed of 30000ms
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
>         at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
>         at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1574)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1531)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>         at java.lang.Thread.run(Thread.java:722)
> Caused by: org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hbase.ClockOutOfSyncException: Server
> blade4,60020,1346451768443 has been rejected; Reported time is too far out
> of sync with master.  Time difference of 496371ms > max allowed of 30000ms
>         at
> org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:193)
>         at
> org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:141)
>         at
> org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:675)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>         at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>         at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>         at $Proxy5.regionServerStartup(Unknown Source)

Hi Tian,

Can you share you configuration files?

Do you have something like that on your hbase-site.xml file?

  <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
    <description>The mode the cluster will be in. Possible values are
      false: standalone and pseudo-distributed setups with managed Zookeeper
      true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
    </description>
  </property>
JM

2012/12/10, tgh <[EMAIL PROTECTED]>:
> Hi
> I try to use hbase, and now ,I have a problem with hbase
> configuration, I use 8node for try, and it seems to work, hadoop,
> zookeeper,
> hbase all boot up, and it could insert into with putAPI ,
> But when I try to use masterIP:60010 to manage it, I find there are only
> one
> regionserver there, and it is localhost:60030, why?
> I have set regionserver file , there are 8 nodes there, and all
> could ssh to each other without passwd,
>
> And I also use hdfsNamenodeIP:50070 to see HDFS, and it seems OK,
> the data have been balanced across 8node, but I wander hbase only have one
> regionserver to work, although when I star/stop regionserver , there are
> 8region server to start and stop,
>
> And I try to put data into hbase, it seems ok at first, but after
> 200million in hbase, it seems really hard to insert more into it, it is
> very
> slow, and I use masterIP:60010 to manage it, I find there are only one
> regionserver there, and it is localhost:60030, why?
>
>
> Could you help me,
>
>
>
> Thank you
> -------------------
> Tian Guanhua
>
>
>
>
>
>
>

Thank you for your reply,
And the configuration file is here,
Could you help me,
Thank you
---------------------------
Tian Guanhua

[root@blade1 conf]# cat regionservers
blade1
blade2
blade3
blade4
blade5
blade6
blade7
blade8
[root@blade1 conf]#
[root@blade1 conf]# vim hbase-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
-->
<configuration>
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://blade1:9000/hbase</value>
    <description>The directory shared by RegionServers.</description>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>blade1,blade2,blade3</value>
</property>
 <property>
    <name>hbase.zookeeper.property.dataDir</name>
   <value>/home/liuxin/zookeeper/data</value>
 </property>
 <property>
  <name>dfs.support.append</name>
   <value>true</value>
 </property>
 <property>
  <name>dfs.datanode.max.xcievers</name>
   <value>4096</value>
   </property>
 <property>
 <name>hbase.master</name>
 <value>blade1:60000</value>
 </property>
</configuration>
-----邮件原件-----
发件人: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] 代表 Jean-Marc Spaggiari
发送时间: 2012年12月10日 20:54
收件人: [EMAIL PROTECTED]
主题: Re: how config multi regionserver, or what is wrong?

Hi Tian,

Can you share you configuration files?

Do you have something like that on your hbase-site.xml file?

  <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
    <description>The mode the cluster will be in. Possible values are
      false: standalone and pseudo-distributed setups with managed Zookeeper
      true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
    </description>
  </property>
JM

2012/12/10, tgh <[EMAIL PROTECTED]>:
> Hi
> I try to use hbase, and now ,I have a problem with hbase
> configuration, I use 8node for try, and it seems to work, hadoop,
> zookeeper, hbase all boot up, and it could insert into with putAPI ,
> But when I try to use masterIP:60010 to manage it, I find there are
> only one regionserver there, and it is localhost:60030, why?
> I have set regionserver file , there are 8 nodes there, and all could
> ssh to each other without passwd,
>
> And I also use hdfsNamenodeIP:50070 to see HDFS, and it seems OK, the
> data have been balanced across 8node, but I wander hbase only have one
> regionserver to work, although when I star/stop regionserver , there
> are 8region server to start and stop,
>
> And I try to put data into hbase, it seems ok at first, but after
> 200million in hbase, it seems really hard to insert more into it, it
> is very slow, and I use masterIP:60010 to manage it, I find there are
> only one regionserver there, and it is localhost:60030, why?
>
>
> Could you help me,
>
>
>
> Thank you
> -------------------
> Tian Guanhua
>
>
>
>
>
>
>

And our hosts is follows

[root@blade1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.76.233 blade1
192.168.76.234 blade2
192.168.76.235 blade3
192.168.76.236 blade4
192.168.76.237 blade5
192.168.76.238 blade6
192.168.76.239 blade7
192.168.76.240 blade8

192.168.76.245 fnode1
192.168.76.246 fnode2
[root@blade1 ~]#

-----邮件原件-----
发件人: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] 代表 tgh
发送时间: 2012年12月11日 9:00
收件人: [EMAIL PROTECTED]
主题: 答复: how config multi regionserver, or what is wrong?

Thank you for your reply,
And the configuration file is here,
Could you help me,
Thank you
---------------------------
Tian Guanhua

[root@blade1 conf]# cat regionservers
blade1
blade2
blade3
blade4
blade5
blade6
blade7
blade8
[root@blade1 conf]#
[root@blade1 conf]# vim hbase-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
-->
<configuration>
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://blade1:9000/hbase</value>
    <description>The directory shared by RegionServers.</description>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>blade1,blade2,blade3</value>
</property>
 <property>
    <name>hbase.zookeeper.property.dataDir</name>
   <value>/home/liuxin/zookeeper/data</value>
 </property>
 <property>
  <name>dfs.support.append</name>
   <value>true</value>
 </property>
 <property>
  <name>dfs.datanode.max.xcievers</name>
   <value>4096</value>
   </property>
 <property>
 <name>hbase.master</name>
 <value>blade1:60000</value>
 </property>
</configuration>
-----邮件原件-----
发件人: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] 代表 Jean-Marc Spaggiari
发送时间: 2012年12月10日 20:54
收件人: [EMAIL PROTECTED]
主题: Re: how config multi regionserver, or what is wrong?

Hi Tian,

Can you share you configuration files?

Do you have something like that on your hbase-site.xml file?

  <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
    <description>The mode the cluster will be in. Possible values are
      false: standalone and pseudo-distributed setups with managed Zookeeper
      true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
    </description>
  </property>
JM

2012/12/10, tgh <[EMAIL PROTECTED]>:
> Hi
> I try to use hbase, and now ,I have a problem with hbase
> configuration, I use 8node for try, and it seems to work, hadoop,
> zookeeper, hbase all boot up, and it could insert into with putAPI ,
> But when I try to use masterIP:60010 to manage it, I find there are
> only one regionserver there, and it is localhost:60030, why?
> I have set regionserver file , there are 8 nodes there, and all could
> ssh to each other without passwd,
>
> And I also use hdfsNamenodeIP:50070 to see HDFS, and it seems OK, the
> data have been balanced across 8node, but I wander hbase only have one
> regionserver to work, although when I star/stop regionserver , there
> are 8region server to start and stop,
>
> And I try to put data into hbase, it seems ok at first, but after
> 200million in hbase, it seems really hard to insert more into it, it
> is very slow, and I use masterIP:60010 to manage it, I find there are
> only one regionserver there, and it is localhost:60030, why?
>
>
> Could you help me,
>
>
>
> Thank you
> -------------------
> Tian Guanhua
>
>
>
>
>
>
>

Meanwhile , log is master , that is, blade1, is like this,  there are some ERRor like this, for  

2012-09-01 06:31:05,558 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=blade2,60020,1346452716636, regionCount=0, userLoad=false
2012-09-01 06:31:05,569 WARN org.apache.hadoop.hbase.master.ServerManager: Server blade4,60020,1346451768443 has been rejected; Reported time is too far out of sync with master.  Time difference of 496371ms > max allowed of 30000ms
2012-09-01 06:31:05,581 WARN org.apache.hadoop.hbase.master.ServerManager: Server blade5,60020,1346452001672 has been rejected; Reported time is too far out of sync with master.  Time difference of 263137ms > max allowed of 30000ms
2012-09-01 06:31:05,583 ERROR org.apache.hadoop.hbase.master.HMaster: Region server serverName=blade4,60020,1346451768443, load=(requests=0, regions=0, usedHeap=142, maxHeap=966) reported a fatal error:
ABORTING region server serverName=blade4,60020,1346451768443, load=(requests=0, regions=0, usedHeap=142, maxHeap=966): Unhandled exception: org.apache.hadoop.hbase.ClockOutOfSyncException: Server blade4,60020,1346451768443 has been rejected; Reported time is too far out of sync with master.  Time difference of 496371ms > max allowed of 30000ms
Cause:
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server blade4,60020,1346451768443 has been rejected; Reported time is too far out of sync with master.  Time difference of 496371ms > max allowed of 30000ms
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1574)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1531)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
        at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server blade4,60020,1346451768443 has been rejected; Reported time is too far out of sync with master.  Time difference of 496371ms > max allowed of 30000ms
        at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:193)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:141)
        at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:675)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
        at $Proxy5.regionServerStartup(Unknown Source)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1570)
        ... 3 more
-----邮件原件-----
发件人: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] 代表 tgh
发送时间: 2012年12月11日 9:59
收件人: [EMAIL PROTECTED]
主题: 答复: how config multi regionserver, or what is wrong?

And our hosts is follows

[root@blade1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

192.168.76.233 blade1
192.168.76.234 blade2
192.168.76.235 blade3
192.168.76.236 blade4
192.168.76.237 blade5
192.168.76.238 blade6
192.168.76.239 blade7
192.168.76.240 blade8

192.168.76.245 fnode1
192.168.76.246 fnode2
[root@blade1 ~]#

发件人: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] 代表 tgh
发送时间: 2012年12月11日 9:00
收件人: [EMAIL PROTECTED]
主题: 答复: how config multi regionserver, or what is wrong?

Thank you for your reply,
And the configuration file is here,
Could you help me,
Thank you
Tian Guanhua

[root@blade1 conf]# cat regionservers
blade1
blade2
blade3
blade4
blade5
blade6
blade7
blade8
[root@blade1 conf]#
[root@blade1 conf]# vim hbase-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
<configuration>
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://blade1:9000/hbase</value>
    <description>The directory shared by RegionServers.</description>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>blade1,blade2,blade3</value>
</property>
 <property>
    <name>hbase.zookeeper.property.dataDir</name>
   <value>/home/liuxin/zookeeper/data</value>
 </property>
 <property>
  <name>dfs.support.append</name>
   <value>true</value>
 </property>
 <property>
  <name>dfs.datanode.max.xcievers</name>
   <value>4096</value>
   </property>
 <property>
 <name>hbase.master</name>
 <value>blade1:60000</value>
 </property>
</configuration>
发件人: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] 代表 Jean-Marc Spaggiari
发送时间: 2012年12月10日 20:54
收件人: [EMAIL PROTECTED]he

Your nodes are too much out of sync with time.
> Reported time is too far out of sync with master.  Time difference of 496371ms > max allowed of 30000ms
You need to set up time synchronization service for your cluster.

Thank you!

Sincerely,
Leonid Fedotov
Hortonworks support team
On Dec 10, 2012, at 6:17 PM, tgh wrote:

> Meanwhile , log is master , that is, blade1, is like this,  there are some ERRor like this, for  
>
> 2012-09-01 06:31:05,558 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=blade2,60020,1346452716636, regionCount=0, userLoad=false
> 2012-09-01 06:31:05,569 WARN org.apache.hadoop.hbase.master.ServerManager: Server blade4,60020,1346451768443 has been rejected; Reported time is too far out of sync with master.  Time difference of 496371ms > max allowed of 30000ms
> 2012-09-01 06:31:05,581 WARN org.apache.hadoop.hbase.master.ServerManager: Server blade5,60020,1346452001672 has been rejected; Reported time is too far out of sync with master.  Time difference of 263137ms > max allowed of 30000ms
> 2012-09-01 06:31:05,583 ERROR org.apache.hadoop.hbase.master.HMaster: Region server serverName=blade4,60020,1346451768443, load=(requests=0, regions=0, usedHeap=142, maxHeap=966) reported a fatal error:
> ABORTING region server serverName=blade4,60020,1346451768443, load=(requests=0, regions=0, usedHeap=142, maxHeap=966): Unhandled exception: org.apache.hadoop.hbase.ClockOutOfSyncException: Server blade4,60020,1346451768443 has been rejected; Reported time is too far out of sync with master.  Time difference of 496371ms > max allowed of 30000ms
> Cause:
> org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server blade4,60020,1346451768443 has been rejected; Reported time is too far out of sync with master.  Time difference of 496371ms > max allowed of 30000ms
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
>        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
>        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1574)
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1531)
>        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>        at java.lang.Thread.run(Thread.java:722)
> Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server blade4,60020,1346451768443 has been rejected; Reported time is too far out of sync with master.  Time difference of 496371ms > max allowed of 30000ms
>        at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:193)
>        at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:141)
>        at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:675)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:601)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>        at $Proxy5.regionServerStartup(Unknown Source)