Hello,

I just saw this in my five-node, three-regionserver cluster test. The
regionserver crashed with this error. Could this be related to some recent
changes involving ZK? Alternatively, this could be a concurrency issue of
its own.

2011-11-21 01:30:15,188 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
<hostname>,60020,1321867814890: Initialization of RS failed.  Hence
aborting RS.
java.util.ConcurrentModificationException
        at java.util.Hashtable$Enumerator.next(Hashtable.java:1031)
        at
org.apache.hadoop.conf.Configuration.iterator(Configuration.java:1042)
        at
org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps(ZKConfig.java:75)
        at
org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:245)
        at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:144)
        at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:124)
        at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1262)
        at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:568)
        at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:559)
        at
org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:183)
        at
org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:177)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:575)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:534)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:642)
        at java.lang.Thread.run(Thread.java:619)

Thanks,
--Mikhail

Were you using 0.92 or 0.89 release ?

Thanks

On Nov 21, 2011, at 1:48 AM, Mikhail Bautin <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I just saw this in my five-node, three-regionserver cluster test. The
> regionserver crashed with this error. Could this be related to some recent
> changes involving ZK? Alternatively, this could be a concurrency issue of
> its own.
>
> 2011-11-21 01:30:15,188 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
> <hostname>,60020,1321867814890: Initialization of RS failed.  Hence
> aborting RS.
> java.util.ConcurrentModificationException
>        at java.util.Hashtable$Enumerator.next(Hashtable.java:1031)
>        at
> org.apache.hadoop.conf.Configuration.iterator(Configuration.java:1042)
>        at
> org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps(ZKConfig.java:75)
>        at
> org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:245)
>        at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:144)
>        at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:124)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1262)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:568)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:559)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:183)
>        at
> org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:177)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:575)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:534)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:642)
>        at java.lang.Thread.run(Thread.java:619)
>
> Thanks,
> --Mikhail

I'd say its a concurrency issue on its own, because we didn't touch ZKConfig, but indeed the timing is suspect.

I opened HBASE-4835 for this.

   - Andy

----- Original Message -----
> From: Mikhail Bautin <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc:
> Sent: Monday, November 21, 2011 1:48 AM
> Subject: regionserver crash due to ZK initialization error
>
> Hello,
>
> I just saw this in my five-node, three-regionserver cluster test. The
> regionserver crashed with this error. Could this be related to some recent
> changes involving ZK? Alternatively, this could be a concurrency issue of
> its own.
>
> 2011-11-21 01:30:15,188 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
> <hostname>,60020,1321867814890: Initialization of RS failed.  Hence
> aborting RS.
> java.util.ConcurrentModificationException
>         at java.util.Hashtable$Enumerator.next(Hashtable.java:1031)
>         at
> org.apache.hadoop.conf.Configuration.iterator(Configuration.java:1042)
>         at
> org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps(ZKConfig.java:75)
>         at
> org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:245)
>         at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:144)
>         at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:124)
>         at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1262)
>         at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:568)
>         at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:559)
>         at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:183)
>         at
> org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:177)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:575)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:534)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:642)
>         at java.lang.Thread.run(Thread.java:619)
>
> Thanks,
> --Mikhail
>