|
|
-
unstable secure zookeeper
Francis Liu 2012-02-17, 01:03
Hi,
I have 0.92-security installed. I'm hitting intermittent problems starting
the regionservers because of intermittent zookeeper connection failures.
Because of this not all my region servers startup after "start
regionservers". This also sometimes happens on the master server.
On the regionserver the error would look like:
2012-02-16 02:57:28,086 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
*snip*,60020,1329361047462: Unexpected exception during initialization,
aborting
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode NoAuth for /hbase/shutdown
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1131)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZ
ooKeeper.java:295)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:518)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:494)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeT
racker.java:77)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HReg
ionServer.java:561)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitializ
ation(HRegionServer.java:524)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:6
25)
at java.lang.Thread.run(Thread.java:619)
If the server does successfully startup things run fine. Prior to this I
had .92 without security running fine. Any ideas on what could be causing
this?
-Francis
-
Re: unstable secure zookeeper
Andrew Purtell 2012-02-17, 01:23
Do you see messages earlier in the log about the ZooKeeper client failing to authenticate? Any other ZooKeeper client messages?
Server side, what do you see in the ZooKeeper logs?
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: Francis Liu <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc:
> Sent: Thursday, February 16, 2012 5:03 PM
> Subject: unstable secure zookeeper
>
> Hi,
>
> I have 0.92-security installed. I'm hitting intermittent problems starting
> the regionservers because of intermittent zookeeper connection failures.
> Because of this not all my region servers startup after "start
> regionservers". This also sometimes happens on the master server.
>
> On the regionserver the error would look like:
>
> 2012-02-16 02:57:28,086 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
> *snip*,60020,1329361047462: Unexpected exception during initialization,
> aborting
> org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode > NoAuth for /hbase/shutdown
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1131)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZ
> ooKeeper.java:295)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:518)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:494)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeT
> racker.java:77)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HReg
> ionServer.java:561)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitializ
> ation(HRegionServer.java:524)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:6
> 25)
> at java.lang.Thread.run(Thread.java:619)
>
>
>
> If the server does successfully startup things run fine. Prior to this I
> had .92 without security running fine. Any ideas on what could be causing
> this?
>
> -Francis
>
-
Re: unstable secure zookeeper
shashwat shriparv 2012-02-17, 07:49
check if 127.0.1.1 is there in your hosts file if it is there either remove
it or make it 127.0.0.1 and see if it solved the problem of starting region
server.
On Fri, Feb 17, 2012 at 6:53 AM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
> Do you see messages earlier in the log about the ZooKeeper client failing
> to authenticate? Any other ZooKeeper client messages?
>
> Server side, what do you see in the ZooKeeper logs?
>
>
> Best regards,
>
>
> - Andy
>
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
>
>
> ----- Original Message -----
> > From: Francis Liu <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]
> > Cc:
> > Sent: Thursday, February 16, 2012 5:03 PM
> > Subject: unstable secure zookeeper
> >
> > Hi,
> >
> > I have 0.92-security installed. I'm hitting intermittent problems
> starting
> > the regionservers because of intermittent zookeeper connection failures.
> > Because of this not all my region servers startup after "start
> > regionservers". This also sometimes happens on the master server.
> >
> > On the regionserver the error would look like:
> >
> > 2012-02-16 02:57:28,086 FATAL
> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
> server
> > *snip*,60020,1329361047462: Unexpected exception during initialization,
> > aborting
> > org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode > > NoAuth for /hbase/shutdown
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1131)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZ
> > ooKeeper.java:295)
> > at
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:518)
> > at
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:494)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeT
> > racker.java:77)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HReg
> > ionServer.java:561)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitializ
> > ation(HRegionServer.java:524)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:6
> > 25)
> > at java.lang.Thread.run(Thread.java:619)
> >
> >
> >
> > If the server does successfully startup things run fine. Prior to this I
> > had .92 without security running fine. Any ideas on what could be causing
> > this?
> >
> > -Francis
> >
>
--
Shashwat Shriparv
-
unstable secure zookeeper
Francis Liu 2012-02-18, 00:27
This see my message popup in the list. Resending....
Hi,
I have 0.92-security installed. I'm hitting intermittent problems starting
the regionservers because of intermittent zookeeper connection failures.
Because of this not all my region servers startup after "start
regionservers". This also sometimes happens on the master server.
On the regionserver the error would look like:
2012-02-16 02:57:28,086 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
*snip*,60020,1329361047462: Unexpected exception during initialization,
aborting
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode NoAuth for /hbase/shutdown
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1131)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZ
ooKeeper.java:295)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:518)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:494)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeT
racker.java:77)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HReg
ionServer.java:561)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitializ
ation(HRegionServer.java:524)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:6
25)
at java.lang.Thread.run(Thread.java:619)
If the server does successfully startup things run fine. Prior to this I
had .92 without security running fine. Any ideas on what could be causing
this?
-Francis
|
|
