Greetings Accumulators,

I'm having a few challenges getting a fully distributed accumulo 1.3.5
cluster working well. (Yes, I'll be moving to 1.4.0 soon).

Accumulo Overview (/status) reports "Master is Down" yet the process is
running on the Master per jps. Port 9999 is not bound though. There was
some talk of this on the list but the discussion did not help my situation.

Also in Accumulo Tablet Servers (/tservers), none of my TSERVERs is listed.
I have verified that both tserver and logger processes are running on all
the slaves.

Everything uses host names so this is not a localhost issues.

The last line of the master_.....log file is:
(timestamp) [master.EventCoordinator]  INFO : State changed from HAVE_LOCK
to WAIT_FOR_TSERVERS.

The last entries of the tserver......log files are:
(timestamp) [cache.LruBlockCache]  INFO : Cache Stats: ......

There is a Failed to load native map library exceptions just above the
Cashe Stats lines.

Any ideas where to go from here? Thank you so very much in advance for any
help you can provide.

David

The master in 1.3 has a small bug, in that it is not responsive to
monitoring if there are no tablet servers, so that's why it is showing down.

The LRU stats are printed in a background thread... ignore them.

Is your zookeeper setting in accumulo-site.xml still set to localhost?

-Eric

On Mon, May 21, 2012 at 3:06 PM, David Arsenault <[EMAIL PROTECTED]>wrote:

> Greetings Accumulators,
>
> I'm having a few challenges getting a fully distributed accumulo 1.3.5
> cluster working well. (Yes, I'll be moving to 1.4.0 soon).
>
> Accumulo Overview (/status) reports "Master is Down" yet the process is
> running on the Master per jps. Port 9999 is not bound though. There was
> some talk of this on the list but the discussion did not help my situation.
>
> Also in Accumulo Tablet Servers (/tservers), none of my TSERVERs is listed.
> I have verified that both tserver and logger processes are running on all
> the slaves.
>
> Everything uses host names so this is not a localhost issues.
>
> The last line of the master_.....log file is:
> (timestamp) [master.EventCoordinator]  INFO : State changed from HAVE_LOCK
> to WAIT_FOR_TSERVERS.
>
> The last entries of the tserver......log files are:
> (timestamp) [cache.LruBlockCache]  INFO : Cache Stats: ......
>
> There is a Failed to load native map library exceptions just above the
> Cashe Stats lines.
>
> Any ideas where to go from here? Thank you so very much in advance for any
> help you can provide.
>
> David
>

No, my accumulo-site.xml points to my 3 ZKs which are verified as
running properly.
> Is your zookeeper setting in accumulo-site.xml still set to localhost?
On Mon, May 21, 2012 at 3:06 PM, David Arsenault <[EMAIL PROTECTED]>wrote:

> Greetings Accumulators,
>
> I'm having a few challenges getting a fully distributed accumulo 1.3.5
> cluster working well. (Yes, I'll be moving to 1.4.0 soon).
>
> Accumulo Overview (/status) reports "Master is Down" yet the process is
> running on the Master per jps. Port 9999 is not bound though. There was
> some talk of this on the list but the discussion did not help my situation.
>
> Also in Accumulo Tablet Servers (/tservers), none of my TSERVERs is
> listed. I have verified that both tserver and logger processes are running
> on all the slaves.
>
> Everything uses host names so this is not a localhost issues.
>
> The last line of the master_.....log file is:
> (timestamp) [master.EventCoordinator]  INFO : State changed from HAVE_LOCK
> to WAIT_FOR_TSERVERS.
>
> The last entries of the tserver......log files are:
> (timestamp) [cache.LruBlockCache]  INFO : Cache Stats: ......
>
> There is a Failed to load native map library exceptions just above the
> Cashe Stats lines.
>
> Any ideas where to go from here? Thank you so very much in advance for any
> help you can provide.
>
> David
>

Any chance that none of your logger processes are up because the
write-ahead log directory doesn't exist?

-Eric

On Tue, May 22, 2012 at 12:40 PM, David Arsenault <[EMAIL PROTECTED]>wrote:

> No, my accumulo-site.xml points to my 3 ZKs which are verified as
> running properly.
>
>
> > Is your zookeeper setting in accumulo-site.xml still set to localhost?
>
>
> On Mon, May 21, 2012 at 3:06 PM, David Arsenault <[EMAIL PROTECTED]
> >wrote:
>
> > Greetings Accumulators,
> >
> > I'm having a few challenges getting a fully distributed accumulo 1.3.5
> > cluster working well. (Yes, I'll be moving to 1.4.0 soon).
> >
> > Accumulo Overview (/status) reports "Master is Down" yet the process is
> > running on the Master per jps. Port 9999 is not bound though. There was
> > some talk of this on the list but the discussion did not help my
> situation.
> >
> > Also in Accumulo Tablet Servers (/tservers), none of my TSERVERs is
> > listed. I have verified that both tserver and logger processes are
> running
> > on all the slaves.
> >
> > Everything uses host names so this is not a localhost issues.
> >
> > The last line of the master_.....log file is:
> > (timestamp) [master.EventCoordinator]  INFO : State changed from
> HAVE_LOCK
> > to WAIT_FOR_TSERVERS.
> >
> > The last entries of the tserver......log files are:
> > (timestamp) [cache.LruBlockCache]  INFO : Cache Stats: ......
> >
> > There is a Failed to load native map library exceptions just above the
> > Cashe Stats lines.
> >
> > Any ideas where to go from here? Thank you so very much in advance for
> any
> > help you can provide.
> >
> > David
> >
>

All loggers are running and the walog directory exists on each node.

Thank you
-David

On Tue, May 22, 2012 at 12:44 PM, Eric Newton <[EMAIL PROTECTED]> wrote:

> Any chance that none of your logger processes are up because the
> write-ahead log directory doesn't exist?
>
> -Eric
>
> On Tue, May 22, 2012 at 12:40 PM, David Arsenault <[EMAIL PROTECTED]
> >wrote:
>
> > No, my accumulo-site.xml points to my 3 ZKs which are verified as
> > running properly.
> >
> >
> > > Is your zookeeper setting in accumulo-site.xml still set to localhost?
> >
> >
> > On Mon, May 21, 2012 at 3:06 PM, David Arsenault <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Greetings Accumulators,
> > >
> > > I'm having a few challenges getting a fully distributed accumulo 1.3.5
> > > cluster working well. (Yes, I'll be moving to 1.4.0 soon).
> > >
> > > Accumulo Overview (/status) reports "Master is Down" yet the process is
> > > running on the Master per jps. Port 9999 is not bound though. There was
> > > some talk of this on the list but the discussion did not help my
> > situation.
> > >
> > > Also in Accumulo Tablet Servers (/tservers), none of my TSERVERs is
> > > listed. I have verified that both tserver and logger processes are
> > running
> > > on all the slaves.
> > >
> > > Everything uses host names so this is not a localhost issues.
> > >
> > > The last line of the master_.....log file is:
> > > (timestamp) [master.EventCoordinator]  INFO : State changed from
> > HAVE_LOCK
> > > to WAIT_FOR_TSERVERS.
> > >
> > > The last entries of the tserver......log files are:
> > > (timestamp) [cache.LruBlockCache]  INFO : Cache Stats: ......
> > >
> > > There is a Failed to load native map library exceptions just above the
> > > Cashe Stats lines.
> > >
> > > Any ideas where to go from here? Thank you so very much in advance for
> > any
> > > help you can provide.
> > >
> > > David
> > >
> >
>

David,

Were you able to figure this one out? Are you still looking for help?

Adam
On Tue, May 22, 2012 at 1:00 PM, David Arsenault <[EMAIL PROTECTED]>wrote:

> All loggers are running and the walog directory exists on each node.
>
> Thank you
> -David
>
> On Tue, May 22, 2012 at 12:44 PM, Eric Newton <[EMAIL PROTECTED]>
> wrote:
>
> > Any chance that none of your logger processes are up because the
> > write-ahead log directory doesn't exist?
> >
> > -Eric
> >
> > On Tue, May 22, 2012 at 12:40 PM, David Arsenault <[EMAIL PROTECTED]
> > >wrote:
> >
> > > No, my accumulo-site.xml points to my 3 ZKs which are verified as
> > > running properly.
> > >
> > >
> > > > Is your zookeeper setting in accumulo-site.xml still set to
> localhost?
> > >
> > >
> > > On Mon, May 21, 2012 at 3:06 PM, David Arsenault <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Greetings Accumulators,
> > > >
> > > > I'm having a few challenges getting a fully distributed accumulo
> 1.3.5
> > > > cluster working well. (Yes, I'll be moving to 1.4.0 soon).
> > > >
> > > > Accumulo Overview (/status) reports "Master is Down" yet the process
> is
> > > > running on the Master per jps. Port 9999 is not bound though. There
> was
> > > > some talk of this on the list but the discussion did not help my
> > > situation.
> > > >
> > > > Also in Accumulo Tablet Servers (/tservers), none of my TSERVERs is
> > > > listed. I have verified that both tserver and logger processes are
> > > running
> > > > on all the slaves.
> > > >
> > > > Everything uses host names so this is not a localhost issues.
> > > >
> > > > The last line of the master_.....log file is:
> > > > (timestamp) [master.EventCoordinator]  INFO : State changed from
> > > HAVE_LOCK
> > > > to WAIT_FOR_TSERVERS.
> > > >
> > > > The last entries of the tserver......log files are:
> > > > (timestamp) [cache.LruBlockCache]  INFO : Cache Stats: ......
> > > >
> > > > There is a Failed to load native map library exceptions just above
> the
> > > > Cashe Stats lines.
> > > >
> > > > Any ideas where to go from here? Thank you so very much in advance
> for
> > > any
> > > > help you can provide.
> > > >
> > > > David
> > > >
> > >
> >
>

Adam,

Never resolved issue with 1.3.5. I am finishing up my 1.4.0 install
tomorrow which I hope resolves the issues. An amazing number of bugs were
closed with the 1.4.0 release!

I'll write if I still have the same issues with 1.4.0.

Thank you,
David

On Tue, May 29, 2012 at 4:34 PM, Adam Fuchs <[EMAIL PROTECTED]> wrote:

> David,
>
> Were you able to figure this one out? Are you still looking for help?
>
> Adam
>
>
> On Tue, May 22, 2012 at 1:00 PM, David Arsenault <[EMAIL PROTECTED]
> >wrote:
>
> > All loggers are running and the walog directory exists on each node.
> >
> > Thank you
> > -David
> >
> > On Tue, May 22, 2012 at 12:44 PM, Eric Newton <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Any chance that none of your logger processes are up because the
> > > write-ahead log directory doesn't exist?
> > >
> > > -Eric
> > >
> > > On Tue, May 22, 2012 at 12:40 PM, David Arsenault <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > No, my accumulo-site.xml points to my 3 ZKs which are verified as
> > > > running properly.
> > > >
> > > >
> > > > > Is your zookeeper setting in accumulo-site.xml still set to
> > localhost?
> > > >
> > > >
> > > > On Mon, May 21, 2012 at 3:06 PM, David Arsenault <
> [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Greetings Accumulators,
> > > > >
> > > > > I'm having a few challenges getting a fully distributed accumulo
> > 1.3.5
> > > > > cluster working well. (Yes, I'll be moving to 1.4.0 soon).
> > > > >
> > > > > Accumulo Overview (/status) reports "Master is Down" yet the
> process
> > is
> > > > > running on the Master per jps. Port 9999 is not bound though. There
> > was
> > > > > some talk of this on the list but the discussion did not help my
> > > > situation.
> > > > >
> > > > > Also in Accumulo Tablet Servers (/tservers), none of my TSERVERs is
> > > > > listed. I have verified that both tserver and logger processes are
> > > > running
> > > > > on all the slaves.
> > > > >
> > > > > Everything uses host names so this is not a localhost issues.
> > > > >
> > > > > The last line of the master_.....log file is:
> > > > > (timestamp) [master.EventCoordinator]  INFO : State changed from
> > > > HAVE_LOCK
> > > > > to WAIT_FOR_TSERVERS.
> > > > >
> > > > > The last entries of the tserver......log files are:
> > > > > (timestamp) [cache.LruBlockCache]  INFO : Cache Stats: ......
> > > > >
> > > > > There is a Failed to load native map library exceptions just above
> > the
> > > > > Cashe Stats lines.
> > > > >
> > > > > Any ideas where to go from here? Thank you so very much in advance
> > for
> > > > any
> > > > > help you can provide.
> > > > >
> > > > > David
> > > > >
> > > >
> > >
> >
>