Hi All,

Is there a trick to running a single, standalone hbase on a single ec2  
instance?  I have hbase running locally as standalone and again on a  
separate testing machine in the office and my java client can talk to  
them both just fine.  If I setup an ec2 instance with the same configs  
as the testing machine my client can't connect - meaning it attempts  
to connect and then hangs with no further logging.  If I turn off  
hbase on the ec2 instance and try to connect I get the usual  
ConnectionException: Connection refused.  When hbase is running I can  
connect to the admin console on :60010 just fine, and I've confirmed  
that there are no blocked ports facing me.

I suspect this has something to with ip addresses, specifically the  
elastic ip story of the external ip not being known to the instance  
(where it instead has a 10.x.x.x address), but fiddling hasn't helped.

Any thoughts?

Thanks,
Oliver

Could it be due to security groups on EC2 - are your EC2 ports open
for HBase connections?

Thanks and Regards,
Sonal
www.meghsoft.com
http://in.linkedin.com/in/sonalgoyal

On Fri, Jun 18, 2010 at 11:11 PM, Oliver Meyn
<[EMAIL PROTECTED]> wrote:
> Hi All,
>
> Is there a trick to running a single, standalone hbase on a single ec2
> instance?  I have hbase running locally as standalone and again on a
> separate testing machine in the office and my java client can talk to them
> both just fine.  If I setup an ec2 instance with the same configs as the
> testing machine my client can't connect - meaning it attempts to connect and
> then hangs with no further logging.  If I turn off hbase on the ec2 instance
> and try to connect I get the usual ConnectionException: Connection refused.
>  When hbase is running I can connect to the admin console on :60010 just
> fine, and I've confirmed that there are no blocked ports facing me.
>
> I suspect this has something to with ip addresses, specifically the elastic
> ip story of the external ip not being known to the instance (where it
> instead has a 10.x.x.x address), but fiddling hasn't helped.
>
> Any thoughts?
>
> Thanks,
> Oliver
>
>

The ports are open, and I see in the zookeeper log:

2010-06-18 18:10:07,676 INFO  
org.apache.zookeeper.server.NIOServerCnxn: Connected to /206.x.x.x:
43800 lastZxid 0
2010-06-18 18:10:07,677 INFO  
org.apache.zookeeper.server.NIOServerCnxn: Creating new session  
0x1294c33a2100004
2010-06-18 18:10:07,678 INFO  
org.apache.zookeeper.server.NIOServerCnxn: Finished init of  
0x1294c33a2100004 valid:true

where the 206.x.x.x is my office ip.  But the master log doesn't move,  
and no data comes back...

Thanks,
Oliver

On 18-Jun-10, at 1:59 PM, Sonal Goyal wrote:

> Could it be due to security groups on EC2 - are your EC2 ports open
> for HBase connections?
>
> Thanks and Regards,
> Sonal
> www.meghsoft.com
> http://in.linkedin.com/in/sonalgoyal
>
>
>
> On Fri, Jun 18, 2010 at 11:11 PM, Oliver Meyn
> <[EMAIL PROTECTED]> wrote:
>> Hi All,
>>
>> Is there a trick to running a single, standalone hbase on a single  
>> ec2
>> instance?  I have hbase running locally as standalone and again on a
>> separate testing machine in the office and my java client can talk  
>> to them
>> both just fine.  If I setup an ec2 instance with the same configs  
>> as the
>> testing machine my client can't connect - meaning it attempts to  
>> connect and
>> then hangs with no further logging.  If I turn off hbase on the ec2  
>> instance
>> and try to connect I get the usual ConnectionException: Connection  
>> refused.
>>  When hbase is running I can connect to the admin console on :60010  
>> just
>> fine, and I've confirmed that there are no blocked ports facing me.
>>
>> I suspect this has something to with ip addresses, specifically the  
>> elastic
>> ip story of the external ip not being known to the instance (where it
>> instead has a 10.x.x.x address), but fiddling hasn't helped.
>>
>> Any thoughts?
>>
>> Thanks,
>> Oliver
>>
>>

--
Oliver Meyn
Software Architect
Zerofootprint Software, Inc.
[EMAIL PROTECTED]
(416) 365-7557 x144

Hi Oliver,

sometimes it happens that daemons are listening on wrong IPs - in this
case it helps to check with something like netstat -nlpt

Cheers,
-- Viktors

On Fri, Jun 18, 2010 at 8:13 PM, Oliver Meyn
<[EMAIL PROTECTED]> wrote:
> The ports are open, and I see in the zookeeper log:
>
> 2010-06-18 18:10:07,676 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Connected to /206.x.x.x:43800 lastZxid 0
> 2010-06-18 18:10:07,677 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Creating new session 0x1294c33a2100004
> 2010-06-18 18:10:07,678 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Finished init of 0x1294c33a2100004 valid:true
>
> where the 206.x.x.x is my office ip.  But the master log doesn't move, and
> no data comes back...
>
> Thanks,
> Oliver
>
> On 18-Jun-10, at 1:59 PM, Sonal Goyal wrote:
>
>> Could it be due to security groups on EC2 - are your EC2 ports open
>> for HBase connections?
>>
>> Thanks and Regards,
>> Sonal
>> www.meghsoft.com
>> http://in.linkedin.com/in/sonalgoyal
>>
>>
>>
>> On Fri, Jun 18, 2010 at 11:11 PM, Oliver Meyn
>> <[EMAIL PROTECTED]> wrote:
>>>
>>> Hi All,
>>>
>>> Is there a trick to running a single, standalone hbase on a single ec2
>>> instance?  I have hbase running locally as standalone and again on a
>>> separate testing machine in the office and my java client can talk to
>>> them
>>> both just fine.  If I setup an ec2 instance with the same configs as the
>>> testing machine my client can't connect - meaning it attempts to connect
>>> and
>>> then hangs with no further logging.  If I turn off hbase on the ec2
>>> instance
>>> and try to connect I get the usual ConnectionException: Connection
>>> refused.
>>>  When hbase is running I can connect to the admin console on :60010 just
>>> fine, and I've confirmed that there are no blocked ports facing me.
>>>
>>> I suspect this has something to with ip addresses, specifically the
>>> elastic
>>> ip story of the external ip not being known to the instance (where it
>>> instead has a 10.x.x.x address), but fiddling hasn't helped.
>>>
>>> Any thoughts?
>>>
>>> Thanks,
>>> Oliver
>>>
>>>
>
> --
> Oliver Meyn
> Software Architect
> Zerofootprint Software, Inc.
> [EMAIL PROTECTED]
> (416) 365-7557 x144
>
>
>
>
>
>

hi Oliver.
I found this problem too.
this probrem came from EC2's system & perhaps Today's HBase can't use that case.

your request from client try connect to hbase by External DNS/IP.
when request reachs EC2, EC2 change Externl IP to EC2's Internal IP.

HBase get request, then return Internal IP which has requested Data is.
so your Client try connect with EC2's innternal IP.

Client & you Expect External IP. but HBase Return Internal IP.
thats the why your Client can't get data.

sorry i'm poor English
thanks.

2010/6/19 Oliver Meyn <[EMAIL PROTECTED]>:
> Hi All,
>
> Is there a trick to running a single, standalone hbase on a single ec2
> instance?  I have hbase running locally as standalone and again on a
> separate testing machine in the office and my java client can talk to them
> both just fine.  If I setup an ec2 instance with the same configs as the
> testing machine my client can't connect - meaning it attempts to connect and
> then hangs with no further logging.  If I turn off hbase on the ec2 instance
> and try to connect I get the usual ConnectionException: Connection refused.
>  When hbase is running I can connect to the admin console on :60010 just
> fine, and I've confirmed that there are no blocked ports facing me.
>
> I suspect this has something to with ip addresses, specifically the elastic
> ip story of the external ip not being known to the instance (where it
> instead has a 10.x.x.x address), but fiddling hasn't helped.
>
> Any thoughts?
>
> Thanks,
> Oliver
>
>

Thanks for your response - I agree that this looks like an ip  
problem.  Further digging suggests I'm lost in the cracks that patch  
1279 tries to solve (https://issues.apache.org/jira/browse/
HBASE-1279).  I think what I'm missing is a working example of how the  
new config should work.

Does anyone with an ec2 cluster (or test machine) that's accessible to  
clients from the outside world care to share their config?  Of most  
interest are the hbase-site.xml entries for *.dns.interface,  
*.dns.nameserver, and hbase.zookeper.quorom.  Naturally the whole file  
would be most illuminating.

Thanks,
Oliver

On 19-Jun-10, at 1:30 AM, 豊月 wrote:

> hi Oliver.
> I found this problem too.
> this probrem came from EC2's system & perhaps Today's HBase can't  
> use that case.
>
> your request from client try connect to hbase by External DNS/IP.
> when request reachs EC2, EC2 change Externl IP to EC2's Internal IP.
>
> HBase get request, then return Internal IP which has requested Data  
> is.
> so your Client try connect with EC2's innternal IP.
>
> Client & you Expect External IP. but HBase Return Internal IP.
> thats the why your Client can't get data.
>
> sorry i'm poor English
> thanks.
>
> 2010/6/19 Oliver Meyn <[EMAIL PROTECTED]>:
>> Hi All,
>>
>> Is there a trick to running a single, standalone hbase on a single  
>> ec2
>> instance?  I have hbase running locally as standalone and again on a
>> separate testing machine in the office and my java client can talk  
>> to them
>> both just fine.  If I setup an ec2 instance with the same configs  
>> as the
>> testing machine my client can't connect - meaning it attempts to  
>> connect and
>> then hangs with no further logging.  If I turn off hbase on the ec2  
>> instance
>> and try to connect I get the usual ConnectionException: Connection  
>> refused.
>> When hbase is running I can connect to the admin console on :60010  
>> just
>> fine, and I've confirmed that there are no blocked ports facing me.
>>
>> I suspect this has something to with ip addresses, specifically the  
>> elastic
>> ip story of the external ip not being known to the instance (where it
>> instead has a 10.x.x.x address), but fiddling hasn't helped.
>>
>> Any thoughts?
>>
>> Thanks,
>> Oliver
>>
>>

Oliver,

Here is our (GumGum's) configuration. Note that we have only one Zookeeper
node (on master) as recommended by Todd Lipcon in a previous thread.
Needless to say that the addresses are not the real addresses.

<configuration>
    <property>
        <name>hbase.rootdir</name>

<value>hdfs://domU-15-35-19-0B-DD-29.compute-1.internal:9000/hbase</value>
    </property>
    <property>
        <name>hbase.master</name>
        <value>domU-15-35-19-0B-DD-29.compute-1.internal:60000</value>
    </property>
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>domU-15-35-19-0B-DD-29.compute-1.internal</value>
    </property>
</configuration>

We have never tried to connect to this hbase form our local machine. All the
HBase clients are in ec2 too. We specify internal ec2 addresses for
everything.

Regards,
Vaibhav
http://aws-musings.com
http://whynosql.com
On Mon, Jun 21, 2010 at 10:47 AM, Oliver Meyn <[EMAIL PROTECTED]
> wrote:

> Thanks for your response - I agree that this looks like an ip problem.
>  Further digging suggests I'm lost in the cracks that patch 1279 tries to
> solve (https://issues.apache.org/jira/browse/HBASE-1279).  I think what
> I'm missing is a working example of how the new config should work.
>
> Does anyone with an ec2 cluster (or test machine) that's accessible to
> clients from the outside world care to share their config?  Of most interest
> are the hbase-site.xml entries for *.dns.interface, *.dns.nameserver, and
> hbase.zookeper.quorom.  Naturally the whole file would be most illuminating.
>
> Thanks,
> Oliver
>
>
> On 19-Jun-10, at 1:30 AM, 豊月 wrote:
>
>  hi Oliver.
>> I found this problem too.
>> this probrem came from EC2's system & perhaps Today's HBase can't use that
>> case.
>>
>> your request from client try connect to hbase by External DNS/IP.
>> when request reachs EC2, EC2 change Externl IP to EC2's Internal IP.
>>
>> HBase get request, then return Internal IP which has requested Data is.
>> so your Client try connect with EC2's innternal IP.
>>
>> Client & you Expect External IP. but HBase Return Internal IP.
>> thats the why your Client can't get data.
>>
>> sorry i'm poor English
>> thanks.
>>
>> 2010/6/19 Oliver Meyn <[EMAIL PROTECTED]>:
>>
>>> Hi All,
>>>
>>> Is there a trick to running a single, standalone hbase on a single ec2
>>> instance?  I have hbase running locally as standalone and again on a
>>> separate testing machine in the office and my java client can talk to
>>> them
>>> both just fine.  If I setup an ec2 instance with the same configs as the
>>> testing machine my client can't connect - meaning it attempts to connect
>>> and
>>> then hangs with no further logging.  If I turn off hbase on the ec2
>>> instance
>>> and try to connect I get the usual ConnectionException: Connection
>>> refused.
>>> When hbase is running I can connect to the admin console on :60010 just
>>> fine, and I've confirmed that there are no blocked ports facing me.
>>>
>>> I suspect this has something to with ip addresses, specifically the
>>> elastic
>>> ip story of the external ip not being known to the instance (where it
>>> instead has a 10.x.x.x address), but fiddling hasn't helped.
>>>
>>> Any thoughts?
>>>
>>> Thanks,
>>> Oliver
>>>
>>>
>>>
>
>
>
>
>
>

I'm pretty sure this is a question that has come up on the list before:

    http://search-hadoop.com/m/sPdqNFAwyg2

Does this address this issue(s)?

> On Mon, Jun 21, 2010 at 10:47 AM, Oliver Meyn <[EMAIL PROTECTED]
> > wrote:
>
> > Thanks for your response - I agree that this looks
> like an ip problem.
> >  Further digging suggests I'm lost in the cracks
> that patch 1279 tries to
> > solve (https://issues.apache.org/jira/browse/HBASE-1279). 
> I think what
> > I'm missing is a working example of how the new config
> should work.
> >
> > Does anyone with an ec2 cluster (or test machine)
> that's accessible to
> > clients from the outside world care to share their
> config?  Of most interest
> > are the hbase-site.xml entries for *.dns.interface,
> *.dns.nameserver, and
> > hbase.zookeper.quorom.  Naturally the whole file
> would be most illuminating.
> >
> > Thanks,
> > Oliver
> >
> >
> > On 19-Jun-10, at 1:30 AM, 豊月 wrote:
> >
> >  hi Oliver.
> >> I found this problem too.
> >> this probrem came from EC2's system & perhaps
> Today's HBase can't use that
> >> case.
> >>
> >> your request from client try connect to hbase by
> External DNS/IP.
> >> when request reachs EC2, EC2 change Externl IP to
> EC2's Internal IP.
> >>
> >> HBase get request, then return Internal IP which
> has requested Data is.
> >> so your Client try connect with EC2's innternal
> IP.
> >>
> >> Client & you Expect External IP. but HBase
> Return Internal IP.
> >> thats the why your Client can't get data.
> >>
> >> sorry i'm poor English
> >> thanks.
> >>
> >> 2010/6/19 Oliver Meyn <[EMAIL PROTECTED]>:
> >>
> >>> Hi All,
> >>>
> >>> Is there a trick to running a single,
> standalone hbase on a single ec2
> >>> instance?  I have hbase running locally
> as standalone and again on a
> >>> separate testing machine in the office and my
> java client can talk to
> >>> them
> >>> both just fine.  If I setup an ec2
> instance with the same configs as the
> >>> testing machine my client can't connect -
> meaning it attempts to connect
> >>> and
> >>> then hangs with no further logging.  If I
> turn off hbase on the ec2
> >>> instance
> >>> and try to connect I get the usual
> ConnectionException: Connection
> >>> refused.
> >>> When hbase is running I can connect to the
> admin console on :60010 just
> >>> fine, and I've confirmed that there are no
> blocked ports facing me.
> >>>
> >>> I suspect this has something to with ip
> addresses, specifically the
> >>> elastic
> >>> ip story of the external ip not being known to
> the instance (where it
> >>> instead has a 10.x.x.x address), but fiddling
> hasn't helped.
> >>>
> >>> Any thoughts?
> >>>
> >>> Thanks,
> >>> Oliver
> >>>
> >>>
> >>>
> >
> >
> >
> >
> >
> >
>

Thank you Andrew - for some reason my digging didn't turn up that  
link, but I had come to the same conclusions as to the actual problem,  
and had conceded that my client (even in testing) would need to live  
in the cloud in order to use internal amazon ip/dns.  That's a drag,  
but is ultimately how the production env wil be setup, so not a big  
deal.

Thanks to all who replied,
Oliver
ps for posterity, the hbase version i'm using is 0.20.4

On 21-Jun-10, at 6:01 PM, Andrew Purtell wrote:

> I'm pretty sure this is a question that has come up on the list  
> before:
>
>    http://search-hadoop.com/m/sPdqNFAwyg2
>
> Does this address this issue(s)?
>
>> On Mon, Jun 21, 2010 at 10:47 AM, Oliver Meyn <[EMAIL PROTECTED]
>>> wrote:
>>
>>> Thanks for your response - I agree that this looks
>> like an ip problem.
>>>   Further digging suggests I'm lost in the cracks
>> that patch 1279 tries to
>>> solve (https://issues.apache.org/jira/browse/HBASE-1279).
>> I think what
>>> I'm missing is a working example of how the new config
>> should work.
>>>
>>> Does anyone with an ec2 cluster (or test machine)
>> that's accessible to
>>> clients from the outside world care to share their
>> config?  Of most interest
>>> are the hbase-site.xml entries for *.dns.interface,
>> *.dns.nameserver, and
>>> hbase.zookeper.quorom.  Naturally the whole file
>> would be most illuminating.
>>>
>>> Thanks,
>>> Oliver
>>>
>>>
>>> On 19-Jun-10, at 1:30 AM, 豊月 wrote:
>>>
>>>   hi Oliver.
>>>> I found this problem too.
>>>> this probrem came from EC2's system & perhaps
>> Today's HBase can't use that
>>>> case.
>>>>
>>>> your request from client try connect to hbase by
>> External DNS/IP.
>>>> when request reachs EC2, EC2 change Externl IP to
>> EC2's Internal IP.
>>>>
>>>> HBase get request, then return Internal IP which
>> has requested Data is.
>>>> so your Client try connect with EC2's innternal
>> IP.
>>>>
>>>> Client & you Expect External IP. but HBase
>> Return Internal IP.
>>>> thats the why your Client can't get data.
>>>>
>>>> sorry i'm poor English
>>>> thanks.
>>>>
>>>> 2010/6/19 Oliver Meyn <[EMAIL PROTECTED]>:
>>>>
>>>>> Hi All,
>>>>>
>>>>> Is there a trick to running a single,
>> standalone hbase on a single ec2
>>>>> instance?  I have hbase running locally
>> as standalone and again on a
>>>>> separate testing machine in the office and my
>> java client can talk to
>>>>> them
>>>>> both just fine.  If I setup an ec2
>> instance with the same configs as the
>>>>> testing machine my client can't connect -
>> meaning it attempts to connect
>>>>> and
>>>>> then hangs with no further logging.  If I
>> turn off hbase on the ec2
>>>>> instance
>>>>> and try to connect I get the usual
>> ConnectionException: Connection
>>>>> refused.
>>>>> When hbase is running I can connect to the
>> admin console on :60010 just
>>>>> fine, and I've confirmed that there are no
>> blocked ports facing me.
>>>>>
>>>>> I suspect this has something to with ip
>> addresses, specifically the
>>>>> elastic
>>>>> ip story of the external ip not being known to
>> the instance (where it
>>>>> instead has a 10.x.x.x address), but fiddling
>> hasn't helped.
>>>>>
>>>>> Any thoughts?
>>>>>
>>>>> Thanks,
>>>>> Oliver
>>>>>