I'm trying to setup HBase on top of my HDFS filesystem.
I currently have 6 servers but only 3 are configured to run zookeeper and
HBase, for testing purposes and the idea of adding the other servers when
this works.
I'm following this guide http://hbase.apache.org/book/configuration.html to
set it up.

I can paste the entire log in pastebin if you want.
HBase-site.xml : http://pastebin.com/Gb8UFSHj

Errors in the log files:

Zookeeper:
----------

logs/hbase-root-zookeeper-HADOOP.domain.be.log (master)
....
....
2012-03-21 09:58:39,576 INFO org.apache.zookeeper.server.ZooKeeperServer:
Established session 0x36347aae9a0000 with negotiated timeout 180000 for
client /10.11.12.206:43325
2012-03-21 09:58:39,611 WARN org.apache.zookeeper.server.NIOServerCnxn:
caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid
0x36347aae9a0000, likely client has closed socket
    at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
    at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
    at java.lang.Thread.run(Thread.java:722)
2012-03-21 09:58:39,617 INFO org.apache.zookeeper.server.NIOServerCnxn:
Closed socket connection for client /10.11.12.206:43325 which had sessionid
0x36347aae9a0000
....
....

on a znode:
logs/hbase-root-zookeeper-HDP08.domain.be.log
.....
.....
2012-03-21 09:58:31,290 INFO
org.apache.zookeeper.server.quorum.QuorumPeerConfig: Defaulting to majority
quorums
2012-03-21 09:58:31,433 DEBUG org.apache.hadoop.hbase.util.VersionInfo:
preRegister called. Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908,
name=log4j:logger=org.apache.hadoop.hbase.util.VersionInfo
2012-03-21 09:58:31,434 DEBUG org.apache.hadoop.hbase.zookeeper.ZKConfig:
preRegister called. Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908,
name=log4j:logger=org.apache.hadoop.hbase.zookeeper.ZKConfig
2012-03-21 09:58:31,434 DEBUG org.apache.hadoop.hbase.HBaseConfiguration:
preRegister called. Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908,
name=log4j:logger=org.apache.hadoop.hbase.HBaseConfiguration
2012-03-21 09:58:31,435 DEBUG org.apache.hadoop.hbase: preRegister called.
Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908,
name=log4j:logger=org.apache.hadoop.hbase
2012-03-21 09:58:31,435 INFO
org.apache.zookeeper.server.quorum.QuorumPeerMain: Starting quorum peer
2012-03-21 09:58:31,836 INFO org.apache.zookeeper.Login: successfully
logged in.
2012-03-21 09:58:31,842 INFO org.apache.zookeeper.Login: TGT refresh thread
started.
.....
.....
a bit further down the log file:
....
....
2012-03-21 09:58:32,425 INFO
org.apache.zookeeper.server.quorum.LearnerHandler: Sending snapshot last
zxid of peer is 0x1800000012  zxid of leader is 0x1900000000sent zxid of db
as 0x1800000012
2012-03-21 09:58:32,466 WARN org.apache.zookeeper.server.quorum.Leader:
Commiting zxid 0x1900000000 from /10.11.12.208:2888 not first!
2012-03-21 09:58:32,466 WARN org.apache.zookeeper.server.quorum.Leader:
First is 0
2012-03-21 09:58:32,466 INFO org.apache.zookeeper.server.quorum.Leader:
Have quorum of supporters; starting up and setting last processed zxid:
107374182400
2012-03-21 09:58:36,505 INFO
org.apache.zookeeper.server.quorum.QuorumCnxManager: Received connection
request /10.11.12.200:46015
....
....

HBase:
-----
master:
....
....
2012-03-21 09:58:39,162 WARN org.apache.zookeeper.ClientCnxn: Zookeeper
client cannot authenticate using the Client section of the supplied
configuration file: '/home/Stijn/hbase-0.92.0/conf/jaas.conf'. Will
continue connection to Zookeeper server without SASL authentication, if
Zookeeper server allows it.
....
....
2012-03-21 09:58:39,313 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server HDP06.domain.be/10.11.12.206:2222,
sessionid = 0x136347a9db40000, negotiated timeout = 180000
2012-03-21 09:58:39,454 ERROR
org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: Failed construction of Master: class
org.apache.hadoop.hbase.master.HMaster
    at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1588)
    at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:144)
    at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1602)
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException:
KeeperErrorCode = InvalidACL for /hbase
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:121)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778)
    .....
    .....

Hi Stijn,

You seem to be hitting permission issues with /hbase

Caused by: org.apache.zookeeper.KeeperException$InvalidACLException:
KeeperErrorCode = InvalidACL for /hbase
   at org.apache.zookeeper.KeeperException.create(KeeperException.java:121)
   at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778)

What are your dfs user permission settings like? Which user are you running
hbase as?

Best Regards,
Sonal
Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
Nube Technologies <http://www.nubetech.co>

<http://in.linkedin.com/in/sonalgoyal>

On Wed, Mar 21, 2012 at 3:49 PM, Stijn Diependaele <[EMAIL PROTECTED]>wrote:

> I'm trying to setup HBase on top of my HDFS filesystem.
> I currently have 6 servers but only 3 are configured to run zookeeper and
> HBase, for testing purposes and the idea of adding the other servers when
> this works.
> I'm following this guide http://hbase.apache.org/book/configuration.htmlto
> set it up.
>
> I can paste the entire log in pastebin if you want.
> HBase-site.xml : http://pastebin.com/Gb8UFSHj
>
> Errors in the log files:
>
> Zookeeper:
> ----------
>
> logs/hbase-root-zookeeper-HADOOP.domain.be.log (master)
> ....
> ....
> 2012-03-21 09:58:39,576 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Established session 0x36347aae9a0000 with negotiated timeout 180000 for
> client /10.11.12.206:43325
> 2012-03-21 09:58:39,611 WARN org.apache.zookeeper.server.NIOServerCnxn:
> caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x36347aae9a0000, likely client has closed socket
>    at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
>    at
>
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
>    at java.lang.Thread.run(Thread.java:722)
> 2012-03-21 09:58:39,617 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Closed socket connection for client /10.11.12.206:43325 which had
> sessionid
> 0x36347aae9a0000
> ....
> ....
>
> on a znode:
> logs/hbase-root-zookeeper-HDP08.domain.be.log
> .....
> .....
> 2012-03-21 09:58:31,290 INFO
> org.apache.zookeeper.server.quorum.QuorumPeerConfig: Defaulting to majority
> quorums
> 2012-03-21 09:58:31,433 DEBUG org.apache.hadoop.hbase.util.VersionInfo:
> preRegister called. Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908,
> name=log4j:logger=org.apache.hadoop.hbase.util.VersionInfo
> 2012-03-21 09:58:31,434 DEBUG org.apache.hadoop.hbase.zookeeper.ZKConfig:
> preRegister called. Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908,
> name=log4j:logger=org.apache.hadoop.hbase.zookeeper.ZKConfig
> 2012-03-21 09:58:31,434 DEBUG org.apache.hadoop.hbase.HBaseConfiguration:
> preRegister called. Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908,
> name=log4j:logger=org.apache.hadoop.hbase.HBaseConfiguration
> 2012-03-21 09:58:31,435 DEBUG org.apache.hadoop.hbase: preRegister called.
> Server=com.sun.jmx.mbeanserver.JmxMBeanServer@4a1908,
> name=log4j:logger=org.apache.hadoop.hbase
> 2012-03-21 09:58:31,435 INFO
> org.apache.zookeeper.server.quorum.QuorumPeerMain: Starting quorum peer
> 2012-03-21 09:58:31,836 INFO org.apache.zookeeper.Login: successfully
> logged in.
> 2012-03-21 09:58:31,842 INFO org.apache.zookeeper.Login: TGT refresh thread
> started.
> .....
> .....
> a bit further down the log file:
> ....
> ....
> 2012-03-21 09:58:32,425 INFO
> org.apache.zookeeper.server.quorum.LearnerHandler: Sending snapshot last
> zxid of peer is 0x1800000012  zxid of leader is 0x1900000000sent zxid of db
> as 0x1800000012
> 2012-03-21 09:58:32,466 WARN org.apache.zookeeper.server.quorum.Leader:
> Commiting zxid 0x1900000000 from /10.11.12.208:2888 not first!
> 2012-03-21 09:58:32,466 WARN org.apache.zookeeper.server.quorum.Leader:
> First is 0
> 2012-03-21 09:58:32,466 INFO org.apache.zookeeper.server.quorum.Leader:
> Have quorum of supporters; starting up and setting last processed zxid:

Thanks for your reply,

Currently I have no special permissions set for dfs, I think it's all
default.
It's the root that's running the dfs, same for zookeeper it's also the root.
But for some reason the log from the hbase master is stored in Stijn, which
is a normal user, I'm not sure if this has anything to do with the problem.
Normally it should be root that runs it so it should be logged into the
hbase-root-master... file.

logs/hbase-Stijn-master-HADOOP.domain.be.log

Now I've removed the hbase.keytab and krb5.keytab and I made new ones. Now
I'm getting a different error:
And it seems that some problem could be the resolving of the hostname, but
normally that's configured well.

.....
.....
2012-03-22 14:19:28,166 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server HDP06.domain.be/10.11.12.206:2222,
sessionid = 0x1363a8fc39d0001, negotiated timeout = 180000
2012-03-22 14:19:28,194 ERROR
org.apache.zookeeper.client.ZooKeeperSaslClient: An error:
(java.security.PrivilegedActionException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not
found in Kerberos database (7) - UNKNOWN_SERVER)]) occurred when evaluating
Zookeeper Quorum Member's  received SASL token. This may be caused by
Java's being unable to resolve the Zookeeper Quorum Member's hostname
correctly. You may want to try to adding
'-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS
environment. Zookeeper Client will go to AUTH_FAILED state.
2012-03-22 14:19:28,194 ERROR org.apache.zookeeper.ClientCnxn: SASL
authentication with Zookeeper Quorum member failed:
javax.security.sasl.SaslException: An error:
(java.security.PrivilegedActionException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not
found in Kerberos database (7) - UNKNOWN_SERVER)]) occurred when evaluating
Zookeeper Quorum Member's  received SASL token. This may be caused by
Java's being unable to resolve the Zookeeper Quorum Member's hostname
correctly. You may want to try to adding
'-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS
environment. Zookeeper Client will go to AUTH_FAILED state.
2012-03-22 14:19:28,229 ERROR
org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: Failed construction of Master: class
org.apache.hadoop.hbase.master.HMaster
    at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1588)
    at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:144)
    at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1602)
Caused by: org.apache.zookeeper.KeeperException$AuthFailedException:
KeeperErrorCode = AuthFailed for /hbase
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1031)
    at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:189)
    at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:892)
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:159)
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:152)
    at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:264)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
    at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1583)
    ... 5 more
2012/3/21 Sonal Goyal <[EMAIL PROTECTED]>

Your logs still point to a permissions issue on /hbase, so I would start at
resolving that first thing..

org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
   at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1602)
*Caused by: org.apache.zookeeper.KeeperException$AuthFailedException:
KeeperErrorCode = AuthFailed for /hbase
   at org.apache.zookeeper.**KeeperException.creat*e(
KeeperException.java:123)
   at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
Best Regards,
Sonal
Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
Nube Technologies <http://www.nubetech.co>

<http://in.linkedin.com/in/sonalgoyal>

On Thu, Mar 22, 2012 at 6:56 PM, Stijn Diependaele <[EMAIL PROTECTED]>wrote:

> Thanks for your reply,
>
> Currently I have no special permissions set for dfs, I think it's all
> default.
> It's the root that's running the dfs, same for zookeeper it's also the
> root.
> But for some reason the log from the hbase master is stored in Stijn, which
> is a normal user, I'm not sure if this has anything to do with the problem.
> Normally it should be root that runs it so it should be logged into the
> hbase-root-master... file.
>
> logs/hbase-Stijn-master-HADOOP.domain.be.log
>
> Now I've removed the hbase.keytab and krb5.keytab and I made new ones. Now
> I'm getting a different error:
> And it seems that some problem could be the resolving of the hostname, but
> normally that's configured well.
>
> .....
> .....
> 2012-03-22 14:19:28,166 INFO org.apache.zookeeper.ClientCnxn: Session
> establishment complete on server HDP06.domain.be/10.11.12.206:2222,
> sessionid = 0x1363a8fc39d0001, negotiated timeout = 180000
> 2012-03-22 14:19:28,194 ERROR
> org.apache.zookeeper.client.ZooKeeperSaslClient: An error:
> (java.security.PrivilegedActionException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - UNKNOWN_SERVER)]) occurred when evaluating
> Zookeeper Quorum Member's  received SASL token. This may be caused by
> Java's being unable to resolve the Zookeeper Quorum Member's hostname
> correctly. You may want to try to adding
> '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS
> environment. Zookeeper Client will go to AUTH_FAILED state.
> 2012-03-22 14:19:28,194 ERROR org.apache.zookeeper.ClientCnxn: SASL
> authentication with Zookeeper Quorum member failed:
> javax.security.sasl.SaslException: An error:
> (java.security.PrivilegedActionException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - UNKNOWN_SERVER)]) occurred when evaluating
> Zookeeper Quorum Member's  received SASL token. This may be caused by
> Java's being unable to resolve the Zookeeper Quorum Member's hostname
> correctly. You may want to try to adding
> '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS
> environment. Zookeeper Client will go to AUTH_FAILED state.
> 2012-03-22 14:19:28,229 ERROR
> org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMaster
>    at
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1588)
>    at
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:144)
>    at
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
>    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>    at
>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
>    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1602)
> Caused by: org.apache.zookeeper.KeeperException$AuthFailedException:
> KeeperErrorCode = AuthFailed for /hbase

Sorry to dig up my thread again but couldn't my problem be related to this:
https://issues.apache.org/jira/browse/HADOOP-7988
?
since I have this error: GSS initiate failed [Caused by GSSException: No
valid credentials provided (Mechanism level: Server not found in Kerberos
database (7) - UNKNOWN_SERVER)] too.
2012/3/23 Sonal Goyal <[EMAIL PROTECTED]>

> Your logs still point to a permissions issue on /hbase, so I would start at
> resolving that first thing..
>
>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
>   at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1602)
> *Caused by: org.apache.zookeeper.KeeperException$AuthFailedException:
> KeeperErrorCode = AuthFailed for /hbase
>    at org.apache.zookeeper.**KeeperException.creat*e(
> KeeperException.java:123)
>   at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>
>
> Best Regards,
> Sonal
> Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
> Nube Technologies <http://www.nubetech.co>
>
> <http://in.linkedin.com/in/sonalgoyal>
>
>
>
>
>
> On Thu, Mar 22, 2012 at 6:56 PM, Stijn Diependaele <[EMAIL PROTECTED]
> >wrote:
>
> > Thanks for your reply,
> >
> > Currently I have no special permissions set for dfs, I think it's all
> > default.
> > It's the root that's running the dfs, same for zookeeper it's also the
> > root.
> > But for some reason the log from the hbase master is stored in Stijn,
> which
> > is a normal user, I'm not sure if this has anything to do with the
> problem.
> > Normally it should be root that runs it so it should be logged into the
> > hbase-root-master... file.
> >
> > logs/hbase-Stijn-master-HADOOP.domain.be.log
> >
> > Now I've removed the hbase.keytab and krb5.keytab and I made new ones.
> Now
> > I'm getting a different error:
> > And it seems that some problem could be the resolving of the hostname,
> but
> > normally that's configured well.
> >
> > .....
> > .....
> > 2012-03-22 14:19:28,166 INFO org.apache.zookeeper.ClientCnxn: Session
> > establishment complete on server HDP06.domain.be/10.11.12.206:2222,
> > sessionid = 0x1363a8fc39d0001, negotiated timeout = 180000
> > 2012-03-22 14:19:28,194 ERROR
> > org.apache.zookeeper.client.ZooKeeperSaslClient: An error:
> > (java.security.PrivilegedActionException:
> > javax.security.sasl.SaslException: GSS initiate failed [Caused by
> > GSSException: No valid credentials provided (Mechanism level: Server not
> > found in Kerberos database (7) - UNKNOWN_SERVER)]) occurred when
> evaluating
> > Zookeeper Quorum Member's  received SASL token. This may be caused by
> > Java's being unable to resolve the Zookeeper Quorum Member's hostname
> > correctly. You may want to try to adding
> > '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS
> > environment. Zookeeper Client will go to AUTH_FAILED state.
> > 2012-03-22 14:19:28,194 ERROR org.apache.zookeeper.ClientCnxn: SASL
> > authentication with Zookeeper Quorum member failed:
> > javax.security.sasl.SaslException: An error:
> > (java.security.PrivilegedActionException:
> > javax.security.sasl.SaslException: GSS initiate failed [Caused by
> > GSSException: No valid credentials provided (Mechanism level: Server not
> > found in Kerberos database (7) - UNKNOWN_SERVER)]) occurred when
> evaluating
> > Zookeeper Quorum Member's  received SASL token. This may be caused by
> > Java's being unable to resolve the Zookeeper Quorum Member's hostname
> > correctly. You may want to try to adding
> > '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS
> > environment. Zookeeper Client will go to AUTH_FAILED state.
> > 2012-03-22 14:19:28,229 ERROR
> > org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
> > java.lang.RuntimeException: Failed construction of Master: class
> > org.apache.hadoop.hbase.master.HMaster
> >    at
> > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1588)
> >    at
> >
> >
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:144)