Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo, mail # user - Thread "shell" Stuck on IO


Copy link to this message
-
Re: Thread "shell" Stuck on IO
Keith Turner 2012-10-17, 21:08
Is the logger at 192.168.0.130 running.   The stack trace indicates
that the master was attempting to contact the logger at 192.168.0.130
to initiate log recovery.

On Wed, Oct 17, 2012 at 4:58 PM, Ott, Charles H. <[EMAIL PROTECTED]> wrote:
> I am using a VMware ESXi 4.1 server  with Cloudbase(Accumulo)  on RHEL5.
>
> I cannot start with a fresh install because I am somewhat required to use
> the preconfigured image on the vm. (business rules out of my hands)
>
> Unfortunately the support for this preconfigured instance is not available
> and I am tasked with getting it working anyway…
>
>
>
> I am able to log into the shell and view the tables, however if  I attempt
> to create a table or perform a scan, a line return is shown and then it just
> hangs there until finally throwing the following error:
>
> WARN thread “shell” stuck on IO to ssdev:9999:9999 (0) for at least 120044
> ms.
>
>
>
> I did also discover that 9999 is the property: master.port.client in my
> conf/accumulo-site.xml
>
>
>
> There is also an event log that was added to the VM with web based UI
> reporting:
>
> Unable to recover
> 192.168.0.130:11224/b4da830b-8ecb-4868-a480-35a39f4af17a(java.io.IOException:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection timed out)
>
>          java.io.IOException:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection timed out
>
>                  at
> cloudbase.server.tabletserver.log.RemoteLogger.<init>(RemoteLogger.java:75)
>
>                  at
> cloudbase.server.master.CoordinateRecoveryTask$RecoveryJob.startCopy(CoordinateRecoveryTask.java:109)
>
>                  at
> cloudbase.server.master.CoordinateRecoveryTask$RecoveryJob.access$400(CoordinateRecoveryTask.java:93)
>
>                  at
> cloudbase.server.master.CoordinateRecoveryTask.recover(CoordinateRecoveryTask.java:279)
>
>                  at
> cloudbase.server.master.Master$TabletGroupWatcher.run(Master.java:1155)
>
>          Caused by: org.apache.thrift.transport.TTransportException:
> java.net.ConnectException: Connection timed out
>
>                  at
> cloudbase.core.client.impl.ThriftTransportPool.createNewTransport(ThriftTransportPool.java:428)
>
>                  at
> cloudbase.core.client.impl.ThriftTransportPool.getTransport(ThriftTransportPool.java:415)
>
>                  at
> cloudbase.core.client.impl.ThriftTransportPool.getTransport(ThriftTransportPool.java:392)
>
>                  at
> cloudbase.core.util.ThriftUtil.getClient(ThriftUtil.java:58)
>
>                  at
> cloudbase.server.tabletserver.log.RemoteLogger.<init>(RemoteLogger.java:73)
>
>                  ... 4 more
>
>          Caused by: java.net.ConnectException: Connection timed out
>
>                  at sun.nio.ch.Net.connect(Native Method)
>
>                  at
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
>
>                  at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:81)
>
>                  at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:65)
>
>                  at
> cloudbase.core.util.TTimeoutTransport.create(TTimeoutTransport.java:23)
>
>                  at
> cloudbase.core.client.impl.ThriftTransportPool.createNewTransport(ThriftTransportPool.java:426)
>
>                  ... 8 more
>
>
>
>
>
> I have seen posts relating this to the walogs folder not being available,
> but I have checked that and the .lock file is being created automatically.
>
> A #netstat | grep 9999 shows no processes using 9999 before logging into the
> shell… so Im not sure there is a port conflict either.
>
>
>
> Any thoughts on the matter would be greatly appreciated.