Hi,

I'm setting up an Accumulo cluster, most of the nodes are up and running
except for two of the Tablet Server nodes. When either tries to connect,
they throw the fallowing error:

24 15:23:47,613 [tabletserver.TabletServer] WARN : Got loadTablet message
from unauthenticatable user: !SYSTEM
24 15:23:47,613 [tabletserver.TabletServer] FATAL: Got message from a
service with a mismatched configuration. Please ensure a compatible
configuration.
org.apache.accumulo.core.client.AccumuloSecurityException: Error
BAD_CREDENTIALS - Username or Password is Invalid
        at
org.apache.accumulo.server.security.ZKAuthenticator.hasSystemPermission(ZKAuthenticator.java:356)
        at
org.apache.accumulo.server.security.Auditor.hasSystemPermission(Auditor.java:149)
        at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.checkPermission(TabletServer.java:1741)
        at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.loadTablet(TabletServer.java:1797)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
        at $Proxy2.loadTablet(Unknown Source)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$loadTablet.process(TabletClientService.java:2494)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
        at
org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
        at
org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
        at
org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:662)

The Master Node, meanwhile, reports:

24 15:23:30,0088 master:HadoopMaster 1 ERROR unable to get tablet server
status 10.1.10.19:9997[13c64ff2f8c09ef] java.io.IOException: Connection
reset by peer

24 15:23:50,0109 master:HadoopMaster 3 ERROR unable to get tablet server
status 10.1.10.19:9997[13c64ff2f8c09ef] java.net.ConnectException:
Connection refused

24 15:23:50,0110 master:HadoopMaster 1 WARN attempting to stop
10.1.10.19:9997[13c64ff2f8c09ef]

I've confirmed that /accumulo/conf/accumulo-site.xml contains the same
password, user, and secret for all the nodes, including the ones that won't
connect.

Any help would be appreciated!

--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/Error-BAD-CREDENTIALS-when-tablet-server-tries-to-connect-tp2640.html
Sent from the Users mailing list archive at Nabble.com.

"24 15:23:47,613 [tabletserver.TabletServer] FATAL: Got message from a
service with a mismatched configuration. Please ensure a compatible
configuration."

 I believe that the master will not allow tservers to join the cluster if
their configuration is different. Check that the conf and lib directories
are the same across the cluster.

-----Original Message-----
From: Max Metzger [mailto:[EMAIL PROTECTED]]
Sent: Thursday, January 24, 2013 6:37 PM
To: [EMAIL PROTECTED]
Subject: Error BAD_CREDENTIALS when tablet server tries to connect

Hi,

I'm setting up an Accumulo cluster, most of the nodes are up and running
except for two of the Tablet Server nodes. When either tries to connect,
they throw the fallowing error:

24 15:23:47,613 [tabletserver.TabletServer] WARN : Got loadTablet message
from unauthenticatable user: !SYSTEM
24 15:23:47,613 [tabletserver.TabletServer] FATAL: Got message from a
service with a mismatched configuration. Please ensure a compatible
configuration.
org.apache.accumulo.core.client.AccumuloSecurityException: Error
BAD_CREDENTIALS - Username or Password is Invalid
        at
org.apache.accumulo.server.security.ZKAuthenticator.hasSystemPermission(ZKAu
thenticator.java:356)
        at
org.apache.accumulo.server.security.Auditor.hasSystemPermission(Auditor.java
:149)
        at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.che
ckPermission(TabletServer.java:1741)
        at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.loa
dTablet(TabletServer.java:1797)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWra
p.java:59)
        at $Proxy2.loadTablet(Unknown Source)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$l
oadTablet.process(TabletClientService.java:2494)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.p
rocess(TabletClientService.java:2037)
        at
org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerU
tils.java:154)
        at
org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingS
erver.java:631)
        at
org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TSer
verUtils.java:202)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
        at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:662)

The Master Node, meanwhile, reports:

24 15:23:30,0088 master:HadoopMaster 1 ERROR unable to get
tablet server
status 10.1.10.19:9997[13c64ff2f8c09ef] java.io.IOException: Connection
reset by peer

24 15:23:50,0109 master:HadoopMaster 3 ERROR unable to get
tablet server
status 10.1.10.19:9997[13c64ff2f8c09ef] java.net.ConnectException:
Connection refused

24 15:23:50,0110 master:HadoopMaster 1 WARN attempting to
stop
10.1.10.19:9997[13c64ff2f8c09ef]

I've confirmed that /accumulo/conf/accumulo-site.xml contains the same
password, user, and secret for all the nodes, including the ones that won't
connect.

Any help would be appreciated!

--
View this message in context:
http://apache-accumulo.1065345.n5.nabble.com/Error-BAD-CREDENTIALS-when-tabl
et-server-tries-to-connect-tp2640.html
Sent from the Users mailing list archive at Nabble.com.

I've confirmed that the /conf and /lib directories are the same for all
nodes.

Also, I forgot to mention, the two nodes that can no longer connect (and
throw the BAD_CREDENTIALS error) were initially able to connect to the rest
of the accumulo cluster, but at some point were dropped or timed out. This
error occurs now, when I try and reconnect them.

--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/Error-BAD-CREDENTIALS-when-tablet-server-tries-to-connect-tp2640p2642.html
Sent from the Users mailing list archive at Nabble.com.

You'll also need to ensure that any configuration properties starting with
"instance." are configured identically (string compare) across all nodes.
This is because the system password is based on a hash of certain
configuration properties that should be set to the same value throughout a
cluster in order for Accumulo to behave well.

--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Thu, Jan 24, 2013 at 6:36 PM, Max Metzger <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I'm setting up an Accumulo cluster, most of the nodes are up and running
> except for two of the Tablet Server nodes. When either tries to connect,
> they throw the fallowing error:
>
> 24 15:23:47,613 [tabletserver.TabletServer] WARN : Got loadTablet message
> from unauthenticatable user: !SYSTEM
> 24 15:23:47,613 [tabletserver.TabletServer] FATAL: Got message from a
> service with a mismatched configuration. Please ensure a compatible
> configuration.
> org.apache.accumulo.core.client.AccumuloSecurityException: Error
> BAD_CREDENTIALS - Username or Password is Invalid
>         at
>
> org.apache.accumulo.server.security.ZKAuthenticator.hasSystemPermission(ZKAuthenticator.java:356)
>         at
>
> org.apache.accumulo.server.security.Auditor.hasSystemPermission(Auditor.java:149)
>         at
>
> org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.checkPermission(TabletServer.java:1741)
>         at
>
> org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.loadTablet(TabletServer.java:1797)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
>
> org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
>         at $Proxy2.loadTablet(Unknown Source)
>         at
>
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$loadTablet.process(TabletClientService.java:2494)
>         at
>
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>         at
>
> org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>         at
>
> org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>         at
>
> org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>         at java.lang.Thread.run(Thread.java:662)
>
> The Master Node, meanwhile, reports:
>
> 24 15:23:30,0088        master:HadoopMaster     1       ERROR unable to
> get tablet server
> status 10.1.10.19:9997[13c64ff2f8c09ef] java.io.IOException: Connection
> reset by peer
>
> 24 15:23:50,0109        master:HadoopMaster     3       ERROR unable to
> get tablet server
> status 10.1.10.19:9997[13c64ff2f8c09ef] java.net.ConnectException:
> Connection refused
>
> 24 15:23:50,0110        master:HadoopMaster     1       WARN attempting to
> stop
> 10.1.10.19:9997[13c64ff2f8c09ef]
>
> I've confirmed that /accumulo/conf/accumulo-site.xml contains the same
> password, user, and secret for all the nodes, including the ones that won't
> connect.
>
> Any help would be appreciated!
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/Error-BAD-CREDENTIALS-when-tablet-server-tries-to-connect-tp2640.html
> Sent from the Users mailing list archive at Nabble.com.
>

Thanks for your help - that fixed it. The accumulo-site.xml for the affected
nodes were listing the Zookeeper nodes in a different order.

--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/Error-BAD-CREDENTIALS-when-tablet-server-tries-to-connect-tp2640p2644.html
Sent from the Users mailing list archive at Nabble.com.