Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> Thread "shell" Stuck on IO


Copy link to this message
-
Re: Thread "shell" Stuck on IO
Is the logger at 192.168.0.130 running.   The stack trace indicates
that the master was attempting to contact the logger at 192.168.0.130
to initiate log recovery.

On Wed, Oct 17, 2012 at 4:58 PM, Ott, Charles H. <[EMAIL PROTECTED]> wrote:
> I am using a VMware ESXi 4.1 server  with Cloudbase(Accumulo)  on RHEL5.
>
> I cannot start with a fresh install because I am somewhat required to use
> the preconfigured image on the vm. (business rules out of my hands)
>
> Unfortunately the support for this preconfigured instance is not available
> and I am tasked with getting it working anyway…
>
>
>
> I am able to log into the shell and view the tables, however if  I attempt
> to create a table or perform a scan, a line return is shown and then it just
> hangs there until finally throwing the following error:
>
> WARN thread “shell” stuck on IO to ssdev:9999:9999 (0) for at least 120044
> ms.
>
>
>
> I did also discover that 9999 is the property: master.port.client in my
> conf/accumulo-site.xml
>
>
>
> There is also an event log that was added to the VM with web based UI
> reporting:
>
> Unable to recover
> 192.168.0.130:11224/b4da830b-8ecb-4868-a480-35a39f4af17a(java.io.IOException:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection timed out)
>
>          java.io.IOException:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection timed out
>
>                  at
> cloudbase.server.tabletserver.log.RemoteLogger.<init>(RemoteLogger.java:75)
>
>                  at
> cloudbase.server.master.CoordinateRecoveryTask$RecoveryJob.startCopy(CoordinateRecoveryTask.java:109)
>
>                  at
> cloudbase.server.master.CoordinateRecoveryTask$RecoveryJob.access$400(CoordinateRecoveryTask.java:93)
>
>                  at
> cloudbase.server.master.CoordinateRecoveryTask.recover(CoordinateRecoveryTask.java:279)
>
>                  at
> cloudbase.server.master.Master$TabletGroupWatcher.run(Master.java:1155)
>
>          Caused by: org.apache.thrift.transport.TTransportException:
> java.net.ConnectException: Connection timed out
>
>                  at
> cloudbase.core.client.impl.ThriftTransportPool.createNewTransport(ThriftTransportPool.java:428)
>
>                  at
> cloudbase.core.client.impl.ThriftTransportPool.getTransport(ThriftTransportPool.java:415)
>
>                  at
> cloudbase.core.client.impl.ThriftTransportPool.getTransport(ThriftTransportPool.java:392)
>
>                  at
> cloudbase.core.util.ThriftUtil.getClient(ThriftUtil.java:58)
>
>                  at
> cloudbase.server.tabletserver.log.RemoteLogger.<init>(RemoteLogger.java:73)
>
>                  ... 4 more
>
>          Caused by: java.net.ConnectException: Connection timed out
>
>                  at sun.nio.ch.Net.connect(Native Method)
>
>                  at
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
>
>                  at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:81)
>
>                  at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:65)
>
>                  at
> cloudbase.core.util.TTimeoutTransport.create(TTimeoutTransport.java:23)
>
>                  at
> cloudbase.core.client.impl.ThriftTransportPool.createNewTransport(ThriftTransportPool.java:426)
>
>                  ... 8 more
>
>
>
>
>
> I have seen posts relating this to the walogs folder not being available,
> but I have checked that and the .lock file is being created automatically.
>
> A #netstat | grep 9999 shows no processes using 9999 before logging into the
> shell… so Im not sure there is a port conflict either.
>
>
>
> Any thoughts on the matter would be greatly appreciated.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB