Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> HBase Issues (perhaps related to 127.0.0.1)


+
Ratner, Alan S 2012-11-21, 20:01
Copy link to this message
-
Re: HBase Issues (perhaps related to 127.0.0.1)
Hello Alan,

    It's better to keep 127.0.0.1 out of your /etc/hosts and make sure you
have proper DNS resolution as it plays an important role in proper Hbase
functioning. Also add the "hbase.zookeeper.property.clientPort" property in
your hbase-site.xml file and see if it works for you.

Regards,
    Mohammad Tariq

On Thu, Nov 22, 2012 at 1:31 AM, Ratner, Alan S (IS) <[EMAIL PROTECTED]>wrote:

> I'd appreciate any suggestions as to how to get HBase up and running.
>  Right now it dies after a few seconds on all servers.  I am using Hadoop
> 1.0.4, ZooKeeper 3.4.4 and HBase 0.94.2 on Ubuntu.
>
> History: Yesterday I managed to get HBase 0.94.2 working but only after
> removing the 127.0.0.1 line from my /etc/hosts file (and synchronizing my
> clocks).  All was fine until this morning when I realized I could not
> initiate remote log-ins to my servers (using VNC or NX) until I restored
> the 127.0.0.1 line in /etc/hosts.  With that restored I am back to a
> non-working HBase.
>
> With HBase managing ZK I see the following in the HBase Master and ZK
> logs, respectively:
> 2012-11-21 13:40:22,236 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
> ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
>
> 2012-11-21 13:40:22,122 WARN org.apache.zookeeper.server.NIOServerCnxn:
> Exception causing close of session 0x0 due to java.io.IOException:
> ZooKeeperServer not running
>
> At roughly the same time (clocks not perfectly synchronized) I see this in
> a Regionserver log:
> 2012-11-21 13:40:57,727 WARN
> org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException:
> java.lang.SecurityException: Unable to locate a login configuration
> occurred when trying to find JAAS configuration.
> ...
> 2012-11-21 13:40:57,848 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
> ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>
> Logs and configuration follows.
>
> Then I tried managing ZK myself and HBase then fails for seemingly
> different reasons.
> 2012-11-21 14:46:37,320 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/backup-masters/hadoop1,60000,1353527196915 already deleted, and this
> is not a retry
>
> 2012-11-21 14:46:47,483 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unhandled exception. Starting shutdown.
> java.net.ConnectException: Call to hadoop1/127.0.0.1:9000 failed on
> connection exception: java.net.ConnectException: Connection refused
>
> Both HMaster error logs (self-managed and me-managed ZK) mention the
> 127.0.0.1 IP address instead of referring to the server by its name
> (hadoop1) or its true IP address or simply as localhost.
>
> So, start-hbase.sh works OK (HB managing ZK):
> ngc@hadoop1:~/hbase-0.94.2$ bin/start-hbase.sh
> hadoop1: starting zookeeper, logging to
> /tmp/hbase-ngc/logs/hbase-ngc-zookeeper-hadoop1.out
> hadoop2: starting zookeeper, logging to
> /tmp/hbase-ngc/logs/hbase-ngc-zookeeper-hadoop2.out
> hadoop3: starting zookeeper, logging to
> /tmp/hbase-ngc/logs/hbase-ngc-zookeeper-hadoop3.out
> starting master, logging to
> /tmp/hbase-ngc/logs/hbase-ngc-master-hadoop1.out
> hadoop2: starting regionserver, logging to
> /tmp/hbase-ngc/logs/hbase-ngc-regionserver-hadoop2.out
> hadoop6: starting regionserver, logging to
> /tmp/hbase-ngc/logs/hbase-ngc-regionserver-hadoop6.out
> hadoop3: starting regionserver, logging to
> /tmp/hbase-ngc/logs/hbase-ngc-regionserver-hadoop3.out
> hadoop5: starting regionserver, logging to
> /tmp/hbase-ngc/logs/hbase-ngc-regionserver-hadoop5.out
> hadoop4: starting regionserver, logging to
> /tmp/hbase-ngc/logs/hbase-ngc-regionserver-hadoop4.out
>
> I have in hbase-site.xml:
>   <property>
>     <name>hbase.cluster.distributed</name>
>     <value>true</value>
>   </property>
>       <property>
>             <name>hbase.master</name>
+
Ratner, Alan S 2012-11-21, 21:02
+
Michael Segel 2012-11-22, 00:40
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB