Hello, all

It's my understanding that it's best practice to colocate thrift on each of
the regionservers, and use load balancing to spread queries between them.
However, I can't seem to get thrift to bind from the regionservers (only
from the hmaster).

The machines have connectivity between them; regionservers connect and
communicate with the hmaster without any problems.

I'm running Hadoop 0.20.2-cdh3u2, HBase 0.90.4-cdh3u2.

Thrift can run succesfully from the HMaster, but not from any of the
datanode/regionservers

What appears to be the relevant content from
hbase-hbase-thrift-region-1.out below:

12/02/09 19:09:17 INFO client.HConnectionManager$HConnectionImplementation:
getMaster attempt 0 of 10 failed; retrying after sleep of 1000
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
        at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
        at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
        at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
        at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
        at $Proxy4.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
        at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:360)
        at
org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:94)
        at
org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:194)
        at
org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:188)
        at
org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:870)
        at
org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927)
12/02/09 19:09:18 INFO client.HConnectionManager$HConnectionImplementation:
getMaster attempt 1 of 10 failed; retrying after sleep of 1000
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
        at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
        at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
        at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
        at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
        at $Proxy4.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
        at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:360)
        at
org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:94)
        at
org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:194)
        at
org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:188)
        at
org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:870)
        at
org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927)
12/02/09 19:09:19 INFO client.HConnectionManager$HConnectionImplementation:
getMaster attempt 2 of 10 failed; retrying after sleep of 1000
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
        at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
        at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
        at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
        at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
        at $Proxy4.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
        at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:360)
        at
org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:94)
        at
org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:194)
        at
org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:188)
        at
org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:870)
        at
org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927)
12/02/09 19:09:20 INFO client.HConnectionManager$HConnectionImplementation:
getMaster attempt 3 of 10 failed; retrying after sleep of 2000
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at
sun.nio.ch.SocketChannelImpl.finishConnect(So

Connection refused is a java exception that usually means that either
there's nothing listening on the port or it's firewall'd. Is
"master:60000" really what you're expecting to see here? Should be
easy to "telnet master 60000" from that machien to really verify that
it works as expected.

J-D

On Thu, Feb 9, 2012 at 11:19 AM, Tomas Tillery <[EMAIL PROTECTED]> wrote:
> Hello, all
>
> It's my understanding that it's best practice to colocate thrift on each of
> the regionservers, and use load balancing to spread queries between them.
> However, I can't seem to get thrift to bind from the regionservers (only
> from the hmaster).
>
> The machines have connectivity between them; regionservers connect and
> communicate with the hmaster without any problems.
>
> I'm running Hadoop 0.20.2-cdh3u2, HBase 0.90.4-cdh3u2.
>
> Thrift can run succesfully from the HMaster, but not from any of the
> datanode/regionservers
>
> What appears to be the relevant content from
> hbase-hbase-thrift-region-1.out below:
>
> 12/02/09 19:09:17 INFO client.HConnectionManager$HConnectionImplementation:
> getMaster attempt 0 of 10 failed; retrying after sleep of 1000
> java.net.ConnectException: Connection refused
>        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>        at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>        at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>        at $Proxy4.getProtocolVersion(Unknown Source)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:360)
>        at
> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:94)
>        at
> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:194)
>        at
> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:188)
>        at
> org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:870)
>        at
> org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927)
> 12/02/09 19:09:18 INFO client.HConnectionManager$HConnectionImplementation:
> getMaster attempt 1 of 10 failed; retrying after sleep of 1000
> java.net.ConnectException: Connection refused
>        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>        at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>        at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>        at $Proxy4.getProtocolVersion(Unknown Source)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:360)

No, it isn't - as far as I know, it should be aiming for 60010, rather than
60000 (though HBase is also bound to 60000, there's no telnet response
there). There's probably something obvious I'm not managing to see, but I
can't find a way to redirect it to the proper port.
Starting thrift with this install is a matter of invoking
/etc/init.d/hadoop-hbase-thrift. Since it isn't invoked through the shell
script, I'm not sure where to specify the port to connect to.

On Thu, Feb 9, 2012 at 2:27 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> Connection refused is a java exception that usually means that either
> there's nothing listening on the port or it's firewall'd. Is
> "master:60000" really what you're expecting to see here? Should be
> easy to "telnet master 60000" from that machien to really verify that
> it works as expected.
>
> J-D
>
> On Thu, Feb 9, 2012 at 11:19 AM, Tomas Tillery <[EMAIL PROTECTED]>
> wrote:
> > Hello, all
> >
> > It's my understanding that it's best practice to colocate thrift on each
> of
> > the regionservers, and use load balancing to spread queries between them.
> > However, I can't seem to get thrift to bind from the regionservers (only
> > from the hmaster).
> >
> > The machines have connectivity between them; regionservers connect and
> > communicate with the hmaster without any problems.
> >
> > I'm running Hadoop 0.20.2-cdh3u2, HBase 0.90.4-cdh3u2.
> >
> > Thrift can run succesfully from the HMaster, but not from any of the
> > datanode/regionservers
> >
> > What appears to be the relevant content from
> > hbase-hbase-thrift-region-1.out below:
> >
> > 12/02/09 19:09:17 INFO
> client.HConnectionManager$HConnectionImplementation:
> > getMaster attempt 0 of 10 failed; retrying after sleep of 1000
> > java.net.ConnectException: Connection refused
> >        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >        at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> >        at
> >
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> >        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
> >        at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> >        at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> >        at
> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> >        at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> >        at $Proxy4.getProtocolVersion(Unknown Source)
> >        at
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> >        at
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> >        at
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> >        at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:360)
> >        at
> > org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:94)
> >        at
> >
> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:194)
> >        at
> >
> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:188)
> >        at
> > org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:870)
> >        at
> > org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927)
> > 12/02/09 19:09:18 INFO
> client.HConnectionManager$HConnectionImplementation:
> > getMaster attempt 1 of 10 failed; retrying after sleep of 1000
> > java.net.ConnectException: Connection refused
> >        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >        at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> >        at
> >
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> >        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
> >        at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)

60010 is the web ui port by default, 60000 is the port the master uses
for communication. Clients (like thrift) and region servers don't rely
on a configured port because the master already posts it in zookeeper
along with its address (that's how your thrift server found it).

If your region servers are using master:60000 and they don't have any
problem connecting to a port that doesn't respond, we have a bigger
issue here :)

J-D

On Thu, Feb 9, 2012 at 11:38 AM, Tomas Tillery <[EMAIL PROTECTED]> wrote:
> No, it isn't - as far as I know, it should be aiming for 60010, rather than
> 60000 (though HBase is also bound to 60000, there's no telnet response
> there). There's probably something obvious I'm not managing to see, but I
> can't find a way to redirect it to the proper port.
> Starting thrift with this install is a matter of invoking
> /etc/init.d/hadoop-hbase-thrift. Since it isn't invoked through the shell
> script, I'm not sure where to specify the port to connect to.
>
> On Thu, Feb 9, 2012 at 2:27 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:
>
>> Connection refused is a java exception that usually means that either
>> there's nothing listening on the port or it's firewall'd. Is
>> "master:60000" really what you're expecting to see here? Should be
>> easy to "telnet master 60000" from that machien to really verify that
>> it works as expected.
>>
>> J-D
>>
>> On Thu, Feb 9, 2012 at 11:19 AM, Tomas Tillery <[EMAIL PROTECTED]>
>> wrote:
>> > Hello, all
>> >
>> > It's my understanding that it's best practice to colocate thrift on each
>> of
>> > the regionservers, and use load balancing to spread queries between them.
>> > However, I can't seem to get thrift to bind from the regionservers (only
>> > from the hmaster).
>> >
>> > The machines have connectivity between them; regionservers connect and
>> > communicate with the hmaster without any problems.
>> >
>> > I'm running Hadoop 0.20.2-cdh3u2, HBase 0.90.4-cdh3u2.
>> >
>> > Thrift can run succesfully from the HMaster, but not from any of the
>> > datanode/regionservers
>> >
>> > What appears to be the relevant content from
>> > hbase-hbase-thrift-region-1.out below:
>> >
>> > 12/02/09 19:09:17 INFO
>> client.HConnectionManager$HConnectionImplementation:
>> > getMaster attempt 0 of 10 failed; retrying after sleep of 1000
>> > java.net.ConnectException: Connection refused
>> >        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> >        at
>> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> >        at
>> >
>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>> >        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
>> >        at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
>> >        at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
>> >        at
>> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
>> >        at
>> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>> >        at $Proxy4.getProtocolVersion(Unknown Source)
>> >        at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>> >        at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>> >        at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>> >        at
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:360)
>> >        at
>> > org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:94)
>> >        at
>> >
>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:194)
>> >        at
>> >
>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:188)
>> >        at
>> > org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:870)
>> >        at
>> > org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927)

That appears to be the case. Here's an excerpt from the regionserver log on
the same machine:

12/01/24 22:11:44 INFO regionserver.HRegionServer: Attempting connect to
Master server at master:60000
12/01/24 22:11:44 INFO regionserver.HRegionServer: Connected to master at
master:60000
12/01/24 22:11:44 INFO regionserver.HRegionServer: Telling master at
master:60000 that we are up
12/01/24 22:11:44 INFO regionserver.HRegionServer: Master passed us address
to use. Was=region-1:60020, Now=region-1:60020

On Thu, Feb 9, 2012 at 2:43 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> 60010 is the web ui port by default, 60000 is the port the master uses
> for communication. Clients (like thrift) and region servers don't rely
> on a configured port because the master already posts it in zookeeper
> along with its address (that's how your thrift server found it).
>
> If your region servers are using master:60000 and they don't have any
> problem connecting to a port that doesn't respond, we have a bigger
> issue here :)
>
> J-D
>
> On Thu, Feb 9, 2012 at 11:38 AM, Tomas Tillery <[EMAIL PROTECTED]>
> wrote:
> > No, it isn't - as far as I know, it should be aiming for 60010, rather
> than
> > 60000 (though HBase is also bound to 60000, there's no telnet response
> > there). There's probably something obvious I'm not managing to see, but I
> > can't find a way to redirect it to the proper port.
> > Starting thrift with this install is a matter of invoking
> > /etc/init.d/hadoop-hbase-thrift. Since it isn't invoked through the shell
> > script, I'm not sure where to specify the port to connect to.
> >
> > On Thu, Feb 9, 2012 at 2:27 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]
> >wrote:
> >
> >> Connection refused is a java exception that usually means that either
> >> there's nothing listening on the port or it's firewall'd. Is
> >> "master:60000" really what you're expecting to see here? Should be
> >> easy to "telnet master 60000" from that machien to really verify that
> >> it works as expected.
> >>
> >> J-D
> >>
> >> On Thu, Feb 9, 2012 at 11:19 AM, Tomas Tillery <[EMAIL PROTECTED]>
> >> wrote:
> >> > Hello, all
> >> >
> >> > It's my understanding that it's best practice to colocate thrift on
> each
> >> of
> >> > the regionservers, and use load balancing to spread queries between
> them.
> >> > However, I can't seem to get thrift to bind from the regionservers
> (only
> >> > from the hmaster).
> >> >
> >> > The machines have connectivity between them; regionservers connect and
> >> > communicate with the hmaster without any problems.
> >> >
> >> > I'm running Hadoop 0.20.2-cdh3u2, HBase 0.90.4-cdh3u2.
> >> >
> >> > Thrift can run succesfully from the HMaster, but not from any of the
> >> > datanode/regionservers
> >> >
> >> > What appears to be the relevant content from
> >> > hbase-hbase-thrift-region-1.out below:
> >> >
> >> > 12/02/09 19:09:17 INFO
> >> client.HConnectionManager$HConnectionImplementation:
> >> > getMaster attempt 0 of 10 failed; retrying after sleep of 1000
> >> > java.net.ConnectException: Connection refused
> >> >        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >> >        at
> >> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> >> >        at
> >> >
> >>
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> >> >        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
> >> >        at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> >> >        at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> >> >        at
> >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> >> >        at
> >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> >> >        at $Proxy4.getProtocolVersion(Unknown Source)
> >> >        at
> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java

I stand corrected - The region servers were still connected because they
had remained connected. Restarting them resulted in them also receiving a
Connection refused response. Error log follows.

12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Opening socket connection to
server master/192.168.1.1:2181
12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Socket connection established
to master/192.168.1.1:2181, initiating session
12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Session establishment complete
on server master/192.168.1.1:2181, sessionid = 0x13568dc6dfe0005,
negotiated timeout = 40000
12/02/10 20:12:24 INFO ipc.HBaseRpcMetrics: Initializing RPC Metrics with
hostName=HRegionServer, port=60020
12/02/10 20:12:24 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=master:2181 sessionTimeout=180000 watcher=regionserver:60020
12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Opening socket connection to
server master/192.168.1.1:2181
12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Socket connection established
to master/192.168.1.1:2181, initiating session
12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Session establishment complete
on server master/192.168.1.1:2181, sessionid = 0x13568dc6dfe000c,
negotiated timeout = 40000
12/02/10 20:12:24 INFO regionserver.ShutdownHook: Installed shutdown hook
thread: Shutdownhook:regionserver60020
12/02/10 20:12:24 INFO regionserver.MemStoreFlusher:
globalMemStoreLimit=395.0m, globalMemStoreLimitLowMark=345.6m,
maxHeap=987.5m
12/02/10 20:12:24 INFO regionserver.HRegionServer: Runs every 10000000ms
12/02/10 20:12:24 INFO regionserver.HRegionServer: Attempting connect to
Master server at master:60000
12/02/10 20:13:24 WARN regionserver.HRegionServer: Unable to connect to
master. Retrying. Error was:
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
        at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
        at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
        at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
        at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
        at $Proxy6.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1477)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1530)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1514)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:573)
        at java.lang.Thread.run(Thread.java:619)

On Thu, Feb 9, 2012 at 2:48 PM, Tomas Tillery <[EMAIL PROTECTED]>wrote:

> That appears to be the case. Here's an excerpt from the regionserver log
> on the same machine:
>
> 12/01/24 22:11:44 INFO regionserver.HRegionServer: Attempting connect to
> Master server at master:60000
> 12/01/24 22:11:44 INFO regionserver.HRegionServer: Connected to master at
> master:60000
> 12/01/24 22:11:44 INFO regionserver.HRegionServer: Telling master at
> master:60000 that we are up
> 12/01/24 22:11:44 INFO regionserver.HRegionServer: Master passed us
> address to use. Was=region-1:60020, Now=region-1:60020
>
>
> On Thu, Feb 9, 2012 at 2:43 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:
>
>> 60010 is the web ui port by default, 60000 is the port the master uses

So is the master alive?

On Fri, Feb 10, 2012 at 12:21 PM, Tomas Tillery <[EMAIL PROTECTED]> wrote:
> I stand corrected - The region servers were still connected because they
> had remained connected. Restarting them resulted in them also receiving a
> Connection refused response. Error log follows.
>
> 12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Opening socket connection to
> server master/192.168.1.1:2181
> 12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Socket connection established
> to master/192.168.1.1:2181, initiating session
> 12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Session establishment complete
> on server master/192.168.1.1:2181, sessionid = 0x13568dc6dfe0005,
> negotiated timeout = 40000
> 12/02/10 20:12:24 INFO ipc.HBaseRpcMetrics: Initializing RPC Metrics with
> hostName=HRegionServer, port=60020
> 12/02/10 20:12:24 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=master:2181 sessionTimeout=180000 watcher=regionserver:60020
> 12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Opening socket connection to
> server master/192.168.1.1:2181
> 12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Socket connection established
> to master/192.168.1.1:2181, initiating session
> 12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Session establishment complete
> on server master/192.168.1.1:2181, sessionid = 0x13568dc6dfe000c,
> negotiated timeout = 40000
> 12/02/10 20:12:24 INFO regionserver.ShutdownHook: Installed shutdown hook
> thread: Shutdownhook:regionserver60020
> 12/02/10 20:12:24 INFO regionserver.MemStoreFlusher:
> globalMemStoreLimit=395.0m, globalMemStoreLimitLowMark=345.6m,
> maxHeap=987.5m
> 12/02/10 20:12:24 INFO regionserver.HRegionServer: Runs every 10000000ms
> 12/02/10 20:12:24 INFO regionserver.HRegionServer: Attempting connect to
> Master server at master:60000
> 12/02/10 20:13:24 WARN regionserver.HRegionServer: Unable to connect to
> master. Retrying. Error was:
> java.net.ConnectException: Connection refused
>        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>        at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>        at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
>        at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>        at $Proxy6.getProtocolVersion(Unknown Source)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1477)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1530)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1514)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:573)
>        at java.lang.Thread.run(Thread.java:619)
>
> On Thu, Feb 9, 2012 at 2:48 PM, Tomas Tillery <[EMAIL PROTECTED]>wrote:
>
>> That appears to be the case. Here's an excerpt from the regionserver log
>> on the same machine:
>>
>> 12/01/24 22:11:44 INFO regionserver.HRegionServer: Attempting connect to
>> Master server at master:60000
>> 12/01/24 22:11:44 INFO regionserver.HRegionServer: Connected to master at
>> master:60000
>> 12/01/24 22:11:44 INFO regionserver.HRegionServer: Telling master at
>> master:60000 that we are up
>> 12/01/24 22:11:44 INFO regionserver.HRegionServer: Master passed us