I'm trying to run a few tests on the combination and I'm hitting a problem that
I haven't seen before in my setup. This could very well, be a configuration
issue, but at this point, I'm stuck and would love to hear what others
have to say.

Here's the deal: my test is doing this:

-------------------------------------------------------------------------------
import org.apache.hadoop.hbase.HBaseConfiguration;
....
    HBaseAdmin(HBaseConfiguration.create());
-------------------------------------------------------------------------------

and is seeing the following exception:

11/11/02 12:32:20 DEBUG ipc.HBaseRPC: Using
org.apache.hadoop.hbase.ipc.WritableRpcEngine for
org.apache.hadoop.hbase.ipc.HMasterInterface
11/11/02 12:32:20 DEBUG ipc.HBaseClient: The ping interval is60000ms.
11/11/02 12:32:20 INFO
client.HConnectionManager$HConnectionImplementation: getMaster attempt
0 of 10 failed; retrying after sleep of 1000
java.net.UnknownHostException: unknown host: ip-10-118-254-245.ec2.internal
       at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.<init>(HBaseClient.java:232)
       at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:992)
       at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:854)
       at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:149)
       at $Proxy9.getProtocolVersion(Unknown Source)
       at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:182)
       at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:295)
       at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:272)
       at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:324)
       at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:594)
       at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:108)
       at org.apache.bigtop.itest.hbase.util.HBaseTestUtil.getAdmin(HBaseTestUtil.java:68)

The host ip-10-118-254-245.ec2.internal is a valid host name, it is
pingable and DNS lookup'able (I even added all the entries for
all the hosts in the cluster to each hosts' /etc/hosts).

On top of that a standalone hbase shell has no trouble creating tables
and putting data into them.

What can be going wrong?

Thanks,
Roman.

In HBaseClient, its here:

      if (remoteId.getAddress().isUnresolved()) {
        throw new UnknownHostException("unknown host: " +
                                       remoteId.getAddress().getHostName());
      }
getAddress is an InetSocketAddress..... and javadoc for isUnresolved:

  "....true if the hostname couldn't be resolved into an InetAddress."

Its kinda baffling Roman especially if other clients -- shell -- can
get to master fine.

St.Ack
On Wed, Nov 2, 2011 at 10:18 AM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
> I'm trying to run a few tests on the combination and I'm hitting a problem that
> I haven't seen before in my setup. This could very well, be a configuration
> issue, but at this point, I'm stuck and would love to hear what others
> have to say.
>
> Here's the deal: my test is doing this:
>
> -------------------------------------------------------------------------------
> import org.apache.hadoop.hbase.HBaseConfiguration;
> ....
>    HBaseAdmin(HBaseConfiguration.create());
> -------------------------------------------------------------------------------
>
> and is seeing the following exception:
>
> 11/11/02 12:32:20 DEBUG ipc.HBaseRPC: Using
> org.apache.hadoop.hbase.ipc.WritableRpcEngine for
> org.apache.hadoop.hbase.ipc.HMasterInterface
> 11/11/02 12:32:20 DEBUG ipc.HBaseClient: The ping interval is60000ms.
> 11/11/02 12:32:20 INFO
> client.HConnectionManager$HConnectionImplementation: getMaster attempt
> 0 of 10 failed; retrying after sleep of 1000
> java.net.UnknownHostException: unknown host: ip-10-118-254-245.ec2.internal
>       at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.<init>(HBaseClient.java:232)
>       at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:992)
>       at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:854)
>       at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:149)
>       at $Proxy9.getProtocolVersion(Unknown Source)
>       at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:182)
>       at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:295)
>       at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:272)
>       at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:324)
>       at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:594)
>       at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:108)
>       at org.apache.bigtop.itest.hbase.util.HBaseTestUtil.getAdmin(HBaseTestUtil.java:68)
>
> The host ip-10-118-254-245.ec2.internal is a valid host name, it is
> pingable and DNS lookup'able (I even added all the entries for
> all the hosts in the cluster to each hosts' /etc/hosts).
>
> On top of that a standalone hbase shell has no trouble creating tables
> and putting data into them.
>
> What can be going wrong?
>
> Thanks,
> Roman.
>

On Wed, Nov 2, 2011 at 10:42 AM, Stack <[EMAIL PROTECTED]> wrote:
> In HBaseClient, its here:
>
>      if (remoteId.getAddress().isUnresolved()) {
>        throw new UnknownHostException("unknown host: " +
>                                       remoteId.getAddress().getHostName());
>      }
>
>
> getAddress is an InetSocketAddress..... and javadoc for isUnresolved:
>
>  "....true if the hostname couldn't be resolved into an InetAddress."
>
> Its kinda baffling Roman especially if other clients -- shell -- can
> get to master fine.

Got that. Now, here's the odd part. When I connect to the VM with
a debugger and dump the value of
     remoteId.address.hostname
here's what I get:
     "\u0000\u0000ip-10-118-254-245.ec2.internal"

That, of course, explains why such an address can't be resolved,
but do you guys have any idea what could put those \u0000\u0000
in front of the actual hostname?

Thanks,
Roman.

I think this is related to HBASE-4300 / HBASE-4692

Cheers

On Wed, Nov 2, 2011 at 12:22 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:

> On Wed, Nov 2, 2011 at 10:42 AM, Stack <[EMAIL PROTECTED]> wrote:
> > In HBaseClient, its here:
> >
> >      if (remoteId.getAddress().isUnresolved()) {
> >        throw new UnknownHostException("unknown host: " +
> >
> remoteId.getAddress().getHostName());
> >      }
> >
> >
> > getAddress is an InetSocketAddress..... and javadoc for isUnresolved:
> >
> >  "....true if the hostname couldn't be resolved into an InetAddress."
> >
> > Its kinda baffling Roman especially if other clients -- shell -- can
> > get to master fine.
>
> Got that. Now, here's the odd part. When I connect to the VM with
> a debugger and dump the value of
>     remoteId.address.hostname
> here's what I get:
>     "\u0000\u0000ip-10-118-254-245.ec2.internal"
>
> That, of course, explains why such an address can't be resolved,
> but do you guys have any idea what could put those \u0000\u0000
> in front of the actual hostname?
>
> Thanks,
> Roman.
>

Pardon me.
The JIRA should be HBASE-3581.

On Wed, Nov 2, 2011 at 12:33 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> I think this is related to HBASE-4300 / HBASE-4692
>
> Cheers
>
>
> On Wed, Nov 2, 2011 at 12:22 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
>
>> On Wed, Nov 2, 2011 at 10:42 AM, Stack <[EMAIL PROTECTED]> wrote:
>> > In HBaseClient, its here:
>> >
>> >      if (remoteId.getAddress().isUnresolved()) {
>> >        throw new UnknownHostException("unknown host: " +
>> >
>> remoteId.getAddress().getHostName());
>> >      }
>> >
>> >
>> > getAddress is an InetSocketAddress..... and javadoc for isUnresolved:
>> >
>> >  "....true if the hostname couldn't be resolved into an InetAddress."
>> >
>> > Its kinda baffling Roman especially if other clients -- shell -- can
>> > get to master fine.
>>
>> Got that. Now, here's the odd part. When I connect to the VM with
>> a debugger and dump the value of
>>     remoteId.address.hostname
>> here's what I get:
>>     "\u0000\u0000ip-10-118-254-245.ec2.internal"
>>
>> That, of course, explains why such an address can't be resolved,
>> but do you guys have any idea what could put those \u0000\u0000
>> in front of the actual hostname?
>>
>> Thanks,
>> Roman.
>>
>
>

On Wed, Nov 2, 2011 at 12:38 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> On Wed, Nov 2, 2011 at 12:22 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
>>
>>> On Wed, Nov 2, 2011 at 10:42 AM, Stack <[EMAIL PROTECTED]> wrote:
>>> > In HBaseClient, its here:
>>> >
>>> >      if (remoteId.getAddress().isUnresolved()) {
>>> >        throw new UnknownHostException("unknown host: " +
>>> >
>>> remoteId.getAddress().getHostName());
>>> >      }
>>> >
>>> >
>>> > getAddress is an InetSocketAddress..... and javadoc for isUnresolved:
>>> >
>>> >  "....true if the hostname couldn't be resolved into an InetAddress."
>>> >
>>> > Its kinda baffling Roman especially if other clients -- shell -- can
>>> > get to master fine.
>>>
>>> Got that. Now, here's the odd part. When I connect to the VM with
>>> a debugger and dump the value of
>>>     remoteId.address.hostname
>>> here's what I get:
>>>     "\u0000\u0000ip-10-118-254-245.ec2.internal"
>>>
>>> That, of course, explains why such an address can't be resolved,
>>> but do you guys have any idea what could put those \u0000\u0000
>>> in front of the actual hostname?
>>>

hbase-4300 added versioning to the ServerName that is stored up in zk.
 Version is a short on the front of the name.  The 'first' VERSION is
0.  Looking in MasterAddressTracker in 0.92, I see it doing this when
you call masterAddressTracker#getMasterAddress (in HConnectionManager
inside getMaster): ServerName.parseVersionedServerName(bytes);  This
should be stripping those version bytes off the front of the name.
You are up-to-date Roman?

St.Ack

Guys, thanks a million for the help!

It was a combination of a protocol version change and old client
(darn -- I was only behind by a week or so!)

Goes to show how fun it is to track trunk ;-)

Thanks,
Roman.

On Wed, Nov 2, 2011 at 3:11 PM, Stack <[EMAIL PROTECTED]> wrote:
> On Wed, Nov 2, 2011 at 12:38 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>> On Wed, Nov 2, 2011 at 12:22 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
>>>
>>>> On Wed, Nov 2, 2011 at 10:42 AM, Stack <[EMAIL PROTECTED]> wrote:
>>>> > In HBaseClient, its here:
>>>> >
>>>> >      if (remoteId.getAddress().isUnresolved()) {
>>>> >        throw new UnknownHostException("unknown host: " +
>>>> >
>>>> remoteId.getAddress().getHostName());
>>>> >      }
>>>> >
>>>> >
>>>> > getAddress is an InetSocketAddress..... and javadoc for isUnresolved:
>>>> >
>>>> >  "....true if the hostname couldn't be resolved into an InetAddress."
>>>> >
>>>> > Its kinda baffling Roman especially if other clients -- shell -- can
>>>> > get to master fine.
>>>>
>>>> Got that. Now, here's the odd part. When I connect to the VM with
>>>> a debugger and dump the value of
>>>>     remoteId.address.hostname
>>>> here's what I get:
>>>>     "\u0000\u0000ip-10-118-254-245.ec2.internal"
>>>>
>>>> That, of course, explains why such an address can't be resolved,
>>>> but do you guys have any idea what could put those \u0000\u0000
>>>> in front of the actual hostname?
>>>>
>
> hbase-4300 added versioning to the ServerName that is stored up in zk.
>  Version is a short on the front of the name.  The 'first' VERSION is
> 0.  Looking in MasterAddressTracker in 0.92, I see it doing this when
> you call masterAddressTracker#getMasterAddress (in HConnectionManager
> inside getMaster): ServerName.parseVersionedServerName(bytes);  This
> should be stripping those version bytes off the front of the name.
> You are up-to-date Roman?
>
> St.Ack
>

On Wed, Nov 2, 2011 at 5:27 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
> Guys, thanks a million for the help!
>
> It was a combination of a protocol version change and old client
> (darn -- I was only behind by a week or so!)
>
> Goes to show how fun it is to track trunk ;-)
>

You are a hard man Roman.  IIUC, hbase trunk is not all you track.
You must eat nails for breakfast and chew rocket fuel to relax.

St.Ack

>>You must eat nails for breakfast and chew rocket fuel to relax.
Ahhhmm..

On Wed, Nov 2, 2011 at 8:02 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Wed, Nov 2, 2011 at 5:27 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
> > Guys, thanks a million for the help!
> >
> > It was a combination of a protocol version change and old client
> > (darn -- I was only behind by a week or so!)
> >
> > Goes to show how fun it is to track trunk ;-)
> >
>
> You are a hard man Roman.  IIUC, hbase trunk is not all you track.
> You must eat nails for breakfast and chew rocket fuel to relax.
>
> St.Ack
>

On Wed, Nov 2, 2011 at 8:02 PM, Stack <[EMAIL PROTECTED]> wrote:
> On Wed, Nov 2, 2011 at 5:27 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
>> Guys, thanks a million for the help!
>>
>> It was a combination of a protocol version change and old client
>> (darn -- I was only behind by a week or so!)
>>
>> Goes to show how fun it is to track trunk ;-)
>>
>
> You are a hard man Roman.  IIUC, hbase trunk is not all you track.
> You must eat nails for breakfast and chew rocket fuel to relax.

That's exactly the content of my lunch box for ApacheCON! How did you know? ;-)

Thanks,
Roman.