Accumulo, mail # dev - Re: Accumulo looking at /accumulo/UUID/masters/lock, can't convert large hex number to Java long

Accumulo looking at /accumulo/UUID/masters/lock, can't convert large hex number to Java long

Chris Nafster 2013-01-14, 20:34

I have installed Accumulo 1.4.2 on top of Hadoop 0.20 and ZooKeeper 3.4.3 on CentOS 2.6.

I can start Hadoop namenode, datanodes, jobtracker, tasktrackers and 3-node zoo ensemble. I can create things in Hadoop hdfs and zookeeper. I did "accumulo init" without problems.

I can reach the accumulo web interface at http://x.x.x.x:50095/ which shows both tablet servers up and happy. Log files everywhere are free of exceptions.

From "accumulo shell", I typed "createtable foobar". One of the accumulo tabletserver log files shows:
2013-01-14 13:12:46,168 [server.TNonblockingServer] ERROR: Unexpected exception while invoking!
java.lang.NumberFormatException: For input string: "b53c3a3610ce0001"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Long.parseLong(Long.java:422)
at org.apache.accumulo.core.zookeeper.ZooUtil$LockID.<init>(ZooUtil.java:64)
at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.checkPermission(TabletServer.java:1794)
at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.loadTablet(TabletServer.java:1814)
at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)

ZooUtil$LockID.<init>() on line 64 is: "eid = Long.parseLong(sa[1], 16);"

so it's trying to convert b53c3a3610ce0001 to a long.

In a separate test file I did Long.parseLong("b53c3a3610ce0001", 16), which indeed throws NumberFormatException.

Long.MAX_LONG is 9223372036854775807, or 7FFFFFFFFFFFFFFF in hex. b53c3a3610ce0001 is larger than this.

I can't see where this "serializedLID" parameter in ZooUtil comes from, or why it would be > Long.MAX_LONG. Any ideas how to trace this and find out what is happening?

Thanks,
- Rich

Re: Accumulo looking at /accumulo/UUID/masters/lock, can't convert large hex number to Java long

Eric Newton 2013-01-14, 22:34

Wow... I've never seen this before.

The hex digit is the Zookeeper session id of the master. The lock is
verified by the tablet server before it accepts the request to load a
tablet by the master. It is encoded in LiveTServerSet.assignTablet.

I can't imagine how you would get an illegal long value, except if your
libraries didn't match across all your nodes.

-Eric

On Mon, Jan 14, 2013 at 3:34 PM, Chris Nafster <[EMAIL PROTECTED]>wrote:

> I have installed Accumulo 1.4.2 on top of Hadoop 0.20 and ZooKeeper 3.4.3
> on CentOS 2.6.
>
> I can start Hadoop namenode, datanodes, jobtracker, tasktrackers and
> 3-node zoo ensemble. I can create things in Hadoop hdfs and zookeeper. I
> did "accumulo init" without problems.
>
> I can reach the accumulo web interface at http://x.x.x.x:50095/ which
> shows both tablet servers up and happy. Log files everywhere are free of
> exceptions.
>
> From "accumulo shell", I typed "createtable foobar". One of the accumulo
> tabletserver log files shows:
> 2013-01-14 13:12:46,168 [server.TNonblockingServer] ERROR: Unexpected
> exception while invoking!
> java.lang.NumberFormatException: For input string: "b53c3a3610ce0001"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
> at java.lang.Long.parseLong(Long.java:422)
> at
> org.apache.accumulo.core.zookeeper.ZooUtil$LockID.<init>(ZooUtil.java:64)
> at
> org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.checkPermission(TabletServer.java:1794)
> at
> org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.loadTablet(TabletServer.java:1814)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
> at
> org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
> at
> org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
> at
> org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>
> ZooUtil$LockID.<init>() on line 64 is: "eid = Long.parseLong(sa[1], 16);"
>
> so it's trying to convert b53c3a3610ce0001 to a long.
>
> In a separate test file I did Long.parseLong("b53c3a3610ce0001", 16),
> which indeed throws NumberFormatException.
>
> Long.MAX_LONG is 9223372036854775807, or 7FFFFFFFFFFFFFFF in hex.
> b53c3a3610ce0001 is larger than this.
>
> I can't see where this "serializedLID" parameter in ZooUtil comes from, or
> why it would be > Long.MAX_LONG. Any ideas how to trace this and find out
> what is happening?
>
> Thanks,
> - Rich
>
>

Re: Accumulo looking at /accumulo/UUID/masters/lock, can't convert large hex number to Java long

Keith Turner 2013-01-14, 23:09

On Mon, Jan 14, 2013 at 5:34 PM, Eric Newton <[EMAIL PROTECTED]> wrote:
> Wow... I've never seen this before.
>
> The hex digit is the Zookeeper session id of the master. The lock is
> verified by the tablet server before it accepts the request to load a
> tablet by the master. It is encoded in LiveTServerSet.assignTablet.
>
> I can't imagine how you would get an illegal long value, except if your
> libraries didn't match across all your nodes.
>
> -Eric
>

Seems like this may be a bug. I looked at
LiveTServerSet.assignTablet() it eventually calls Long.toHexString().
The javadoc for toHexString() says the following.

   * Returns a string representation of the <code>long</code>
   * argument as an unsigned integer in base 16.

However, the method we are using to parse the Long can not handle
unsigned longs greater than MAX_LONG. There does not seem to be a
method in long that can parse the output of toHexString(). For
example the following will fail, any negative number will fail.

   Long.parseLong(Long.toHexString(-1), 16);

Rich, want to open a bug? If not I can.

Keith

>
>
> On Mon, Jan 14, 2013 at 3:34 PM, Chris Nafster <[EMAIL PROTECTED]>wrote:
>
>> I have installed Accumulo 1.4.2 on top of Hadoop 0.20 and ZooKeeper 3.4.3
>> on CentOS 2.6.
>>
>> I can start Hadoop namenode, datanodes, jobtracker, tasktrackers and
>> 3-node zoo ensemble. I can create things in Hadoop hdfs and zookeeper. I
>> did "accumulo init" without problems.
>>
>> I can reach the accumulo web interface at http://x.x.x.x:50095/ which
>> shows both tablet servers up and happy. Log files everywhere are free of
>> exceptions.
>>
>> From "accumulo shell", I typed "createtable foobar". One of the accumulo
>> tabletserver log files shows:
>> 2013-01-14 13:12:46,168 [server.TNonblockingServer] ERROR: Unexpected
>> exception while invoking!
>> java.lang.NumberFormatException: For input string: "b53c3a3610ce0001"
>> at
>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
>> at java.lang.Long.parseLong(Long.java:422)
>> at
>> org.apache.accumulo.core.zookeeper.ZooUtil$LockID.<init>(ZooUtil.java:64)
>> at
>> org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.checkPermission(TabletServer.java:1794)
>> at
>> org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.loadTablet(TabletServer.java:1814)
>> at
>> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
>> at
>> org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
>> at
>> org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
>> at
>> org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
>>
>> ZooUtil$LockID.<init>() on line 64 is: "eid = Long.parseLong(sa[1], 16);"
>>
>> so it's trying to convert b53c3a3610ce0001 to a long.
>>
>> In a separate test file I did Long.parseLong("b53c3a3610ce0001", 16),
>> which indeed throws NumberFormatException.
>>
>> Long.MAX_LONG is 9223372036854775807, or 7FFFFFFFFFFFFFFF in hex.
>> b53c3a3610ce0001 is larger than this.
>>
>> I can't see where this "serializedLID" parameter in ZooUtil comes from, or
>> why it would be > Long.MAX_LONG. Any ideas how to trace this and find out
>> what is happening?
>>
>> Thanks,
>> - Rich
>>
>>

Re: Accumulo looking at /accumulo/UUID/masters/lock, can't convert large hex number to Java long

Chris Nafster 2013-01-14, 23:28

> Rich, want to open a bug? If not I can.

Done! https://issues.apache.org/jira/browse/ACCUMULO-965. Please edit this to change whatever you feel is important.

To finish up this thread, is there a workaround for now? I.e., can you control or affect the ZooKeeper session ID of the master, or just keep restarting until I get lucky?

- Rich

Re: Accumulo looking at /accumulo/UUID/masters/lock, can't convert large hex number to Java long

Keith Turner 2013-01-14, 23:42

I have never seen this. What version of zookeeper are you running?

On Mon, Jan 14, 2013 at 6:28 PM, Chris Nafster <[EMAIL PROTECTED]> wrote:
>> Rich, want to open a bug? If not I can.
>
> Done! https://issues.apache.org/jira/browse/ACCUMULO-965. Please edit this to change whatever you feel is important.
>
> To finish up this thread, is there a workaround for now? I.e., can you control or affect the ZooKeeper session ID of the master, or just keep restarting until I get lucky?
>
> - Rich
>