I stood up a 4 node Accumulo Cloud.  I wrote a Java interface to it and when I attempt to connect to the table through my interface, I'm getting an intermittent error that only pops up maybe 1 out of every 5 runs of the code.  I have zookeeper running on all 4 nodes but in my accumulo config file, I only mentioned the one on the Accumulo name node.  I don't know if that would be a problem or not.  Also, the zookeeper stuff is all owned by root, while hadoop and Accumulo are owned by a different non-root user.  I built this cluster from the ground up (i.e. didn't use Cloudera, etc.), so it's possible that I missed something.

I'm running Accumulo 1.4.1 with hadoop 1.0.3 and zookeeper 3.3.6.

As you can see from the error below, it is saying that I have an INVALID_INSTANCEID.  However my instance (and credentials) are static final Strings and are not changing.  I haven't had any trouble connecting to the instance via the shell.  Also, the monitor webpage works just fine and shows no errors.

Here is the error below:

Exception in thread "main" org.apache.accumulo.core.client.AccumuloSecurityException: Error INVALID_INSTANCEID - Unknown security exception
                at org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:73)
                at org.apache.accumulo.core.client.impl.ConnectorImpl.<init>(ConnectorImpl.java:75)
                at org.apache.accumulo.core.client.ZooKeeperInstance.getConnector(ZooKeeperInstance.java:218)
                at jhuapl.redd.coremax.shared.accumuloInterface.AccumuloConnector.deleteTable(AccumuloConnector.java:583)
                at jhuapl.redd.coremax.shared.accumuloInterface.TestAccumuloInterface.main(TestAccumuloInterface.java:27)
Caused by: ThriftSecurityException(user:root, code:INVALID_INSTANCEID)
                at org.apache.accumulo.core.client.impl.thrift.ClientService$authenticateUser_result.read(ClientService.java:7755)
                at org.apache.accumulo.core.client.impl.thrift.ClientService$Client.recv_authenticateUser(ClientService.java:415)
                at org.apache.accumulo.core.client.impl.thrift.ClientService$Client.authenticateUser(ClientService.java:387)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:616)
                at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
                at $Proxy0.authenticateUser(Unknown Source)
                at org.apache.accumulo.core.client.impl.ConnectorImpl$1.execute(ConnectorImpl.java:78)
                at org.apache.accumulo.core.client.impl.ConnectorImpl$1.execute(ConnectorImpl.java:75)
                at org.apache.accumulo.core.client.impl.ServerClient.executeRaw(ServerClient.java:108)
                at org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:71)
                ... 4 more

Any help would be greatly appreciated!
Thanks,
Duane

Duane,

The INVALID_INSTANCEID message implies that your client is trying to
contact a server that thinks the instance id is something else. The 1 out
of 5 likelihood probably indicates that one out of the five server
processes (one master and four tablet servers) has some misconfiguration.
When you do a "deleteTable" operation, the client code will get the list of
active servers and pick one at random. This would seem to imply that
zookeeper is configured correctly across your cluster.

Tablet servers read their instance name and ID out of the accumulo
directory in HDFS. I wonder if this could be caused by something like HDFS
not being configured the same on one of the tablet servers, so when it
launches it doesn't get the right instance ID? (I'm thinking out loud here)

Do you see any warning or error messages propagating to your monitor page?

Adam

On Tue, Sep 11, 2012 at 5:25 PM, Cornish, Duane C. <[EMAIL PROTECTED]
> wrote:

> I stood up a 4 node Accumulo Cloud.  I wrote a Java interface to it and
> when I attempt to connect to the table through my interface, I'm getting an
> intermittent error that only pops up maybe 1 out of every 5 runs of the
> code.  I have zookeeper running on all 4 nodes but in my accumulo config
> file, I only mentioned the one on the Accumulo name node.  I don't know if
> that would be a problem or not.  Also, the zookeeper stuff is all owned by
> root, while hadoop and Accumulo are owned by a different non-root user.  I
> built this cluster from the ground up (i.e. didn't use Cloudera, etc.), so
> it's possible that I missed something.
>
> I'm running Accumulo 1.4.1 with hadoop 1.0.3 and zookeeper 3.3.6.
>
> As you can see from the error below, it is saying that I have an
> INVALID_INSTANCEID.  However my instance (and credentials) are static final
> Strings and are not changing.  I haven't had any trouble connecting to the
> instance via the shell.  Also, the monitor webpage works just fine and
> shows no errors.
>
> Here is the error below:
>
> Exception in thread "main"
> org.apache.accumulo.core.client.AccumuloSecurityException: Error
> INVALID_INSTANCEID - Unknown security exception
>                 at
> org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:73)
>                 at
> org.apache.accumulo.core.client.impl.ConnectorImpl.<init>(ConnectorImpl.java:75)
>                 at
> org.apache.accumulo.core.client.ZooKeeperInstance.getConnector(ZooKeeperInstance.java:218)
>                 at
> jhuapl.redd.coremax.shared.accumuloInterface.AccumuloConnector.deleteTable(AccumuloConnector.java:583)
>                 at
> jhuapl.redd.coremax.shared.accumuloInterface.TestAccumuloInterface.main(TestAccumuloInterface.java:27)
> Caused by: ThriftSecurityException(user:root, code:INVALID_INSTANCEID)
>                 at
> org.apache.accumulo.core.client.impl.thrift.ClientService$authenticateUser_result.read(ClientService.java:7755)
>                 at
> org.apache.accumulo.core.client.impl.thrift.ClientService$Client.recv_authenticateUser(ClientService.java:415)
>                 at
> org.apache.accumulo.core.client.impl.thrift.ClientService$Client.authenticateUser(ClientService.java:387)
>                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>                 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>                 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>                 at java.lang.reflect.Method.invoke(Method.java:616)
>                 at
> org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>                 at $Proxy0.authenticateUser(Unknown Source)
>                 at
> org.apache.accumulo.core.client.impl.ConnectorImpl$1.execute(ConnectorImpl.java:78)
>                 at
> org.apache.accumulo.core.client.impl.ConnectorImpl$1.execute(ConnectorImpl.java:75)
>                 at
> org.apache.accumulo.core.client.impl.ServerClient.executeRaw(ServerClient.java:108)

Why are you running four copies of Zookeeper? The recommendation is to
run a prime number of copies (1, 3, 5, etc.). Consider cutting back to
one instance or three.

Duane,

One ZooKeeper host should be more than enough for 5 servers.

Out of curiosity, did you configure zookeeper to be replicated? In your
zoo.cfg, did you create "server.X" entries for each ZooKeeper and create
the myid file in your ZooKeeper dataDir on each host? It sounds like you
initialized 4 unique ZooKeepers, hit one ZooKeeper on init, and then hit
another ZooKeeper when you went to connect.

See http://zookeeper.apache.org/doc/r3.3.6/zookeeperStarted.html,
specifically "Running Replicated ZooKeeper" if you do want to run more
than one ZooKeeper instance.

On 9/11/2012 5:25 PM, Cornish, Duane C. wrote:
> I stood up a 4 node Accumulo Cloud.  I wrote a Java interface to it and when I attempt to connect to the table through my interface, I'm getting an intermittent error that only pops up maybe 1 out of every 5 runs of the code.  I have zookeeper running on all 4 nodes but in my accumulo config file, I only mentioned the one on the Accumulo name node.  I don't know if that would be a problem or not.  Also, the zookeeper stuff is all owned by root, while hadoop and Accumulo are owned by a different non-root user.  I built this cluster from the ground up (i.e. didn't use Cloudera, etc.), so it's possible that I missed something.
>
> I'm running Accumulo 1.4.1 with hadoop 1.0.3 and zookeeper 3.3.6.
>
> As you can see from the error below, it is saying that I have an INVALID_INSTANCEID.  However my instance (and credentials) are static final Strings and are not changing.  I haven't had any trouble connecting to the instance via the shell.  Also, the monitor webpage works just fine and shows no errors.
>
> Here is the error below:
>
> Exception in thread "main" org.apache.accumulo.core.client.AccumuloSecurityException: Error INVALID_INSTANCEID - Unknown security exception
>                  at org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:73)
>                  at org.apache.accumulo.core.client.impl.ConnectorImpl.<init>(ConnectorImpl.java:75)
>                  at org.apache.accumulo.core.client.ZooKeeperInstance.getConnector(ZooKeeperInstance.java:218)
>                  at jhuapl.redd.coremax.shared.accumuloInterface.AccumuloConnector.deleteTable(AccumuloConnector.java:583)
>                  at jhuapl.redd.coremax.shared.accumuloInterface.TestAccumuloInterface.main(TestAccumuloInterface.java:27)
> Caused by: ThriftSecurityException(user:root, code:INVALID_INSTANCEID)
>                  at org.apache.accumulo.core.client.impl.thrift.ClientService$authenticateUser_result.read(ClientService.java:7755)
>                  at org.apache.accumulo.core.client.impl.thrift.ClientService$Client.recv_authenticateUser(ClientService.java:415)
>                  at org.apache.accumulo.core.client.impl.thrift.ClientService$Client.authenticateUser(ClientService.java:387)
>                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>                  at java.lang.reflect.Method.invoke(Method.java:616)
>                  at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>                  at $Proxy0.authenticateUser(Unknown Source)
>                  at org.apache.accumulo.core.client.impl.ConnectorImpl$1.execute(ConnectorImpl.java:78)
>                  at org.apache.accumulo.core.client.impl.ConnectorImpl$1.execute(ConnectorImpl.java:75)
>                  at org.apache.accumulo.core.client.impl.ServerClient.executeRaw(ServerClient.java:108)
>                  at org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:71)
>                  ... 4 more
>
> Any help would be greatly appreciated!
> Thanks,
> Duane
>
>