Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo, mail # user - Thread "shell" Stuck on IO


Copy link to this message
-
Re: Thread "shell" Stuck on IO
Eric Newton 2012-10-18, 11:41
The reference to 192.168.0.130 is in zookeeper or the metadata table.

Unfortunately, this is a known problem with 1.3 and 1.4.  You can't change
your IP addresses.  You can incrementally shutdown servers and change the
IP address one-at-a-time, but not all at once.

If this is a dev system and you don't need the data, the fastest thing to
do is to reset the system and re-load your test data.

If you can't reload your data, you will have to move your data in hdfs,
re-initialize and bulk-import the existing tables.

-Eric
On Wed, Oct 17, 2012 at 5:40 PM, Ott, Charles H. <[EMAIL PROTECTED]>wrote:

> I believe you have already helped me get on the right track...
>
> First, 192.168.0.130 is the IP that the VM came with preconfigured.
> I changed the IP for this new environment in RHEL5 and "most" everything
> seems to be running... however, the fact that it is reporting
> 192.168.0.130 tells me that somewhere in the logger configuration it's
> still using the old IP?
>
> All of the properties files I have looked at specify the hostname, not
> IP... I checked the hosts file and the hostname is resolving the proper
> IP, so that shouldn't be an issue.
>
> When I try to start the logger with:
>
> # ./cloudbase.sh logger
>
>  I see:
> Failed to initialize log service args=[]
>         java.io.IOException: Failed to acquire lock file
>                 at
> cloudbase.server.logger.LogService.<init>(LogService.java:122)
>                 at
> cloudbase.server.logger.LogService.main(LogService.java:83)
>                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>                 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
>                 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>                 at cloudbase.start.Main$1.run(Main.java:73)
>                 at java.lang.Thread.run(Thread.java:662)
>
> -----Original Message-----
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED]] On
> Behalf Of Keith Turner
> Sent: Wednesday, October 17, 2012 5:09 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Thread "shell" Stuck on IO
>
> Is the logger at 192.168.0.130 running.   The stack trace indicates
> that the master was attempting to contact the logger at 192.168.0.130 to
> initiate log recovery.
>
> On Wed, Oct 17, 2012 at 4:58 PM, Ott, Charles H.
> <[EMAIL PROTECTED]> wrote:
> > I am using a VMware ESXi 4.1 server  with Cloudbase(Accumulo)  on
> RHEL5.
> >
> > I cannot start with a fresh install because I am somewhat required to
> > use the preconfigured image on the vm. (business rules out of my
> > hands)
> >
> > Unfortunately the support for this preconfigured instance is not
> > available and I am tasked with getting it working anyway...
> >
> >
> >
> > I am able to log into the shell and view the tables, however if  I
> > attempt to create a table or perform a scan, a line return is shown
> > and then it just hangs there until finally throwing the following
> error:
> >
> > WARN thread "shell" stuck on IO to ssdev:9999:9999 (0) for at least
> > 120044 ms.
> >
> >
> >
> > I did also discover that 9999 is the property: master.port.client in
> > my conf/accumulo-site.xml
> >
> >
> >
> > There is also an event log that was added to the VM with web based UI
> > reporting:
> >
> > Unable to recover
> >
> 192.168.0.130:11224/b4da830b-8ecb-4868-a480-35a39f4af17a(java.io.IOExcep
> tion:
> > org.apache.thrift.transport.TTransportException:
> java.net.ConnectException:
> > Connection timed out)
> >
> >          java.io.IOException:
> > org.apache.thrift.transport.TTransportException:
> java.net.ConnectException:
> > Connection timed out
> >
> >                  at
> > cloudbase.server.tabletserver.log.RemoteLogger.<init>(RemoteLogger.jav
> > a:75)
> >
> >                  at