Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - HMaster shutdown when a DNS address cannot be solved


Copy link to this message
-
Re: HMaster shutdown when a DNS address cannot be solved
Amandeep Khurana 2012-04-08, 23:49
+user
(bcc: dev)

Mikael,

Such questions are better suited for the user mailing list. You'll
find more people talking about issues that they ran into and possibly
get answers to your questions faster.

Hadoop internally using a form of the linux 'hostname' command from
within Java. When servers report into the master, they register with
that hostname. Now, if the hosts cannot be reached from outside
through that name, you'll run into this issue. In other words, you
need a working DNS to get Hadoop/HBase to work properly.  In your
case, there is no way for the FQDN (server16.doman….) to get mapped to
the IP address it seems. You need to fix your host resolution and
restore it to the working state that it was in earlier.

Hope this helps.

-Amandeep
On Sun, Apr 8, 2012 at 11:36 AM, Mikael Sitruk <[EMAIL PROTECTED]> wrote:
>
> Hi devs.
>
> I have a strange situation with my cluster when an address cannot be
> resolved.
> Few days ago I had two entries in a DNS file, so a computer could be found
> either via <host name> or <hostname>.<domain>
> Now the domain entries in the DNS resolving was removed, (so only the short
> name exists) when i try to start the cluster the master fail indicating
> that the server address cannot be resolved see below...
> Any help appreciated???
> BTW this is hadoop-1.0.0 and HBase-0.92.0
>
> 2012-04-08 21:06:45,496 INFO
> org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of
> -ROOT-,,0 at address=server118,60020,1333842630426;
> org.apache.hadoop.hbase.NotServingRegionException: org.apache.had
> oop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
> 2012-04-08 21:06:45,497 INFO
> org.apache.hadoop.hbase.catalog.RootLocationEditor: Unsetting ROOT region
> location in ZooKeeper
> 2012-04-08 21:06:45,969 INFO
> org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED
> event for -ROOT-,,0.70236052 from server119,60020,1333908308851; deleting
> unassigned node
> 2012-04-08 21:06:45,973 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region -ROOT-,,0.70236052 that was online on server119,60020,1333908308851
> 2012-04-08 21:06:45,976 INFO org.apache.hadoop.hbase.master.HMaster: -ROOT-
> assigned=1, rit=false, location=server119,60020,1333908308851
> 2012-04-08 21:06:46,041 FATAL org.apache.hadoop.hbase.master.HMaster:
> Master server abort: loaded coprocessors are: []
> 2012-04-08 21:06:46,042 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unhandled exception. Starting shutdown.
> java.net.UnknownHostException: unknown host: server116.<domain>
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.<init>(HBaseClient.java:227)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1016)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:878)
>        at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
>        at $Proxy12.getProtocolVersion(Unknown Source)
>        at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:183)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1278)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1235)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1222)
>        at
> org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:564)
>        at
> org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:422)