HBase, mail # user - Re: Hbase out of memory error

Hbase out of memory error

Royston Sellman 2012-01-20, 17:20

Trying to run my code (a test of Aggregation Protocol and an MR HBase table
loader) on latest build of 0.92.0 (r1232715) I get an 'old server' warning
(I've seen this before and it's always been non-fatal) then an out of memory
exception then job hangs:

[sshexec] 12/01/20 16:56:48 WARN zookeeper.ClientCnxnSocket: Connected to an
old server; r-o mode will be unavailable

  [sshexec] 12/01/20 16:56:48 INFO zookeeper.ClientCnxn: Session
establishment complete on server namenode/10.0.0.235:2181, sessionid 0x34cda4e5d000e5, negotiated timeout = 40000

  [sshexec] 12/01/20 16:56:49 WARN ipc.HBaseClient: Unexpected exception
receiving call responses

  [sshexec] java.lang.OutOfMemoryError: Java heap space

  [sshexec] at java.lang.reflect.Array.newArray(Native Method)

  [sshexec] at java.lang.reflect.Array.newInstance(Array.java:52)

  [sshexec] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritabl
e.java:542)

  [sshexec] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritabl
e.java:289)

  [sshexec] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
nt.java:593)

  [sshexec] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)

Hbase shell seems to work (I can list and scan my tables).

If I svn roll back to 12 Jan 0.92 and rebuild my code works.

Tried setting export HBASE_HEAPSIZE=1500 but got same error.

Nothing significant in logs.

[Note to Ted Yu: I need to fix this so I can carry on testing on Aggregation
Protocol]

Best Regards,

Royston

Re: Hbase out of memory error

Ted Yu 2012-01-20, 18:45

Royston:
I guess you have seen HBASE-5204. In particular:
>> when a 0.92 server fails to deserialize a 0.90-style RPC, it attempts to
allocate a large buffer because it doesn't read fields of 0.90-style RPCs
properly.

Were your client code compiled with the same version of HBase as what was
running on your cluster ?

Thanks

On Fri, Jan 20, 2012 at 9:20 AM, Royston Sellman <
[EMAIL PROTECTED]> wrote:

> Trying to run my code (a test of Aggregation Protocol and an MR HBase table
> loader) on latest build of 0.92.0 (r1232715) I get an 'old server' warning
> (I've seen this before and it's always been non-fatal) then an out of
> memory
> exception then job hangs:
>
>
>
> [sshexec] 12/01/20 16:56:48 WARN zookeeper.ClientCnxnSocket: Connected to
> an
> old server; r-o mode will be unavailable
>
> [sshexec] 12/01/20 16:56:48 INFO zookeeper.ClientCnxn: Session
> establishment complete on server namenode/10.0.0.235:2181, sessionid > 0x34cda4e5d000e5, negotiated timeout = 40000
>
> [sshexec] 12/01/20 16:56:49 WARN ipc.HBaseClient: Unexpected exception
> receiving call responses
>
> [sshexec] java.lang.OutOfMemoryError: Java heap space
>
> [sshexec] at java.lang.reflect.Array.newArray(Native Method)
>
> [sshexec] at java.lang.reflect.Array.newInstance(Array.java:52)
>
> [sshexec] at
>
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritabl
> e.java:542)
>
> [sshexec] at
>
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritabl
> e.java:289)
>
> [sshexec] at
>
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
> nt.java:593)
>
> [sshexec] at
>
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
>
>
>
> Hbase shell seems to work (I can list and scan my tables).
>
>
>
> If I svn roll back to 12 Jan 0.92 and rebuild my code works.
>
>
>
> Tried setting export HBASE_HEAPSIZE=1500 but got same error.
>
>
>
> Nothing significant in logs.
>
>
>
> [Note to Ted Yu: I need to fix this so I can carry on testing on
> Aggregation
> Protocol]
>
>
>
> Best Regards,
>
> Royston
>
>

Re: Hbase out of memory error

Royston Sellman 2012-01-21, 16:50

Hi Ted,

Yes, I am compiling with the same HBase jars. I wasn't aware of HBASE-5204, thanks, it sounds possible this is my problem. Can you think of anything else I should check?

Just to make sure: I am checking out the code from svn.apache.org/repos/asf/hbase/branches/0.92 Is this the correct branch?

Thanks,
Royston
On 20 Jan 2012, at 18:45, Ted Yu wrote:

> Royston:
> I guess you have seen HBASE-5204. In particular:
>>> when a 0.92 server fails to deserialize a 0.90-style RPC, it attempts to
> allocate a large buffer because it doesn't read fields of 0.90-style RPCs
> properly.
>
> Were your client code compiled with the same version of HBase as what was
> running on your cluster ?
>
> Thanks
>
> On Fri, Jan 20, 2012 at 9:20 AM, Royston Sellman <
> [EMAIL PROTECTED]> wrote:
>
>> Trying to run my code (a test of Aggregation Protocol and an MR HBase table
>> loader) on latest build of 0.92.0 (r1232715) I get an 'old server' warning
>> (I've seen this before and it's always been non-fatal) then an out of
>> memory
>> exception then job hangs:
>>
>>
>>
>> [sshexec] 12/01/20 16:56:48 WARN zookeeper.ClientCnxnSocket: Connected to
>> an
>> old server; r-o mode will be unavailable
>>
>> [sshexec] 12/01/20 16:56:48 INFO zookeeper.ClientCnxn: Session
>> establishment complete on server namenode/10.0.0.235:2181, sessionid >> 0x34cda4e5d000e5, negotiated timeout = 40000
>>
>> [sshexec] 12/01/20 16:56:49 WARN ipc.HBaseClient: Unexpected exception
>> receiving call responses
>>
>> [sshexec] java.lang.OutOfMemoryError: Java heap space
>>
>> [sshexec] at java.lang.reflect.Array.newArray(Native Method)
>>
>> [sshexec] at java.lang.reflect.Array.newInstance(Array.java:52)
>>
>> [sshexec] at
>>
>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritabl
>> e.java:542)
>>
>> [sshexec] at
>>
>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritabl
>> e.java:289)
>>
>> [sshexec] at
>>
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
>> nt.java:593)
>>
>> [sshexec] at
>>
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
>>
>>
>>
>> Hbase shell seems to work (I can list and scan my tables).
>>
>>
>>
>> If I svn roll back to 12 Jan 0.92 and rebuild my code works.
>>
>>
>>
>> Tried setting export HBASE_HEAPSIZE=1500 but got same error.
>>
>>
>>
>> Nothing significant in logs.
>>
>>
>>
>> [Note to Ted Yu: I need to fix this so I can carry on testing on
>> Aggregation
>> Protocol]
>>
>>
>>
>> Best Regards,
>>
>> Royston
>>
>>

Re: Hbase out of memory error

yuzhihong@... 2012-01-21, 16:58

That is the correct branch.

Thanks

On Jan 21, 2012, at 8:50 AM, Royston Sellman <[EMAIL PROTECTED]> wrote:

> Hi Ted,
>
> Yes, I am compiling with the same HBase jars. I wasn't aware of HBASE-5204, thanks, it sounds possible this is my problem. Can you think of anything else I should check?
>
> Just to make sure: I am checking out the code from svn.apache.org/repos/asf/hbase/branches/0.92 Is this the correct branch?
>
> Thanks,
> Royston
>
>
> On 20 Jan 2012, at 18:45, Ted Yu wrote:
>
>> Royston:
>> I guess you have seen HBASE-5204. In particular:
>>>> when a 0.92 server fails to deserialize a 0.90-style RPC, it attempts to
>> allocate a large buffer because it doesn't read fields of 0.90-style RPCs
>> properly.
>>
>> Were your client code compiled with the same version of HBase as what was
>> running on your cluster ?
>>
>> Thanks
>>
>> On Fri, Jan 20, 2012 at 9:20 AM, Royston Sellman <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Trying to run my code (a test of Aggregation Protocol and an MR HBase table
>>> loader) on latest build of 0.92.0 (r1232715) I get an 'old server' warning
>>> (I've seen this before and it's always been non-fatal) then an out of
>>> memory
>>> exception then job hangs:
>>>
>>>
>>>
>>> [sshexec] 12/01/20 16:56:48 WARN zookeeper.ClientCnxnSocket: Connected to
>>> an
>>> old server; r-o mode will be unavailable
>>>
>>> [sshexec] 12/01/20 16:56:48 INFO zookeeper.ClientCnxn: Session
>>> establishment complete on server namenode/10.0.0.235:2181, sessionid =
>>> 0x34cda4e5d000e5, negotiated timeout = 40000
>>>
>>> [sshexec] 12/01/20 16:56:49 WARN ipc.HBaseClient: Unexpected exception
>>> receiving call responses
>>>
>>> [sshexec] java.lang.OutOfMemoryError: Java heap space
>>>
>>> [sshexec] at java.lang.reflect.Array.newArray(Native Method)
>>>
>>> [sshexec] at java.lang.reflect.Array.newInstance(Array.java:52)
>>>
>>> [sshexec] at
>>>
>>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritabl
>>> e.java:542)
>>>
>>> [sshexec] at
>>>
>>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritabl
>>> e.java:289)
>>>
>>> [sshexec] at
>>>
>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
>>> nt.java:593)
>>>
>>> [sshexec] at
>>>
>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
>>>
>>>
>>>
>>> Hbase shell seems to work (I can list and scan my tables).
>>>
>>>
>>>
>>> If I svn roll back to 12 Jan 0.92 and rebuild my code works.
>>>
>>>
>>>
>>> Tried setting export HBASE_HEAPSIZE=1500 but got same error.
>>>
>>>
>>>
>>> Nothing significant in logs.
>>>
>>>
>>>
>>> [Note to Ted Yu: I need to fix this so I can carry on testing on
>>> Aggregation
>>> Protocol]
>>>
>>>
>>>
>>> Best Regards,
>>>
>>> Royston
>>>
>>>
>

Re: Hbase out of memory error

Royston Sellman 2012-01-21, 17:11

So should I try applying Benoit Sigoure's patch for HBASE-5204? Will this patch be in the 0.92 branch soon?

Cheers,
Royston

On 21 Jan 2012, at 16:58, [EMAIL PROTECTED] wrote:

> That is the correct branch.
>
> Thanks
>
>
>
> On Jan 21, 2012, at 8:50 AM, Royston Sellman <[EMAIL PROTECTED]> wrote:
>
>> Hi Ted,
>>
>> Yes, I am compiling with the same HBase jars. I wasn't aware of HBASE-5204, thanks, it sounds possible this is my problem. Can you think of anything else I should check?
>>
>> Just to make sure: I am checking out the code from svn.apache.org/repos/asf/hbase/branches/0.92 Is this the correct branch?
>>
>> Thanks,
>> Royston
>>
>>
>> On 20 Jan 2012, at 18:45, Ted Yu wrote:
>>
>>> Royston:
>>> I guess you have seen HBASE-5204. In particular:
>>>>> when a 0.92 server fails to deserialize a 0.90-style RPC, it attempts to
>>> allocate a large buffer because it doesn't read fields of 0.90-style RPCs
>>> properly.
>>>
>>> Were your client code compiled with the same version of HBase as what was
>>> running on your cluster ?
>>>
>>> Thanks
>>>
>>> On Fri, Jan 20, 2012 at 9:20 AM, Royston Sellman <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Trying to run my code (a test of Aggregation Protocol and an MR HBase table
>>>> loader) on latest build of 0.92.0 (r1232715) I get an 'old server' warning
>>>> (I've seen this before and it's always been non-fatal) then an out of
>>>> memory
>>>> exception then job hangs:
>>>>
>>>>
>>>>
>>>> [sshexec] 12/01/20 16:56:48 WARN zookeeper.ClientCnxnSocket: Connected to
>>>> an
>>>> old server; r-o mode will be unavailable
>>>>
>>>> [sshexec] 12/01/20 16:56:48 INFO zookeeper.ClientCnxn: Session
>>>> establishment complete on server namenode/10.0.0.235:2181, sessionid >>>> 0x34cda4e5d000e5, negotiated timeout = 40000
>>>>
>>>> [sshexec] 12/01/20 16:56:49 WARN ipc.HBaseClient: Unexpected exception
>>>> receiving call responses
>>>>
>>>> [sshexec] java.lang.OutOfMemoryError: Java heap space
>>>>
>>>> [sshexec] at java.lang.reflect.Array.newArray(Native Method)
>>>>
>>>> [sshexec] at java.lang.reflect.Array.newInstance(Array.java:52)
>>>>
>>>> [sshexec] at
>>>>
>>>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritabl
>>>> e.java:542)
>>>>
>>>> [sshexec] at
>>>>
>>>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritabl
>>>> e.java:289)
>>>>
>>>> [sshexec] at
>>>>
>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
>>>> nt.java:593)
>>>>
>>>> [sshexec] at
>>>>
>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
>>>>
>>>>
>>>>
>>>> Hbase shell seems to work (I can list and scan my tables).
>>>>
>>>>
>>>>
>>>> If I svn roll back to 12 Jan 0.92 and rebuild my code works.
>>>>
>>>>
>>>>
>>>> Tried setting export HBASE_HEAPSIZE=1500 but got same error.
>>>>
>>>>
>>>>
>>>> Nothing significant in logs.
>>>>
>>>>
>>>>
>>>> [Note to Ted Yu: I need to fix this so I can carry on testing on
>>>> Aggregation
>>>> Protocol]
>>>>
>>>>
>>>>
>>>> Best Regards,
>>>>
>>>> Royston
>>>>
>>>>
>>

Re: Hbase out of memory error

yuzhihong@... 2012-01-21, 17:13

Benoit's patches are already in 0.92

Thanks
On Jan 21, 2012, at 9:11 AM, Royston Sellman <[EMAIL PROTECTED]> wrote:

> So should I try applying Benoit Sigoure's patch for HBASE-5204? Will this patch be in the 0.92 branch soon?
>
> Cheers,
> Royston
>
>
>
> On 21 Jan 2012, at 16:58, [EMAIL PROTECTED] wrote:
>
>> That is the correct branch.
>>
>> Thanks
>>
>>
>>
>> On Jan 21, 2012, at 8:50 AM, Royston Sellman <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Ted,
>>>
>>> Yes, I am compiling with the same HBase jars. I wasn't aware of HBASE-5204, thanks, it sounds possible this is my problem. Can you think of anything else I should check?
>>>
>>> Just to make sure: I am checking out the code from svn.apache.org/repos/asf/hbase/branches/0.92 Is this the correct branch?
>>>
>>> Thanks,
>>> Royston
>>>
>>>
>>> On 20 Jan 2012, at 18:45, Ted Yu wrote:
>>>
>>>> Royston:
>>>> I guess you have seen HBASE-5204. In particular:
>>>>>> when a 0.92 server fails to deserialize a 0.90-style RPC, it attempts to
>>>> allocate a large buffer because it doesn't read fields of 0.90-style RPCs
>>>> properly.
>>>>
>>>> Were your client code compiled with the same version of HBase as what was
>>>> running on your cluster ?
>>>>
>>>> Thanks
>>>>
>>>> On Fri, Jan 20, 2012 at 9:20 AM, Royston Sellman <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> Trying to run my code (a test of Aggregation Protocol and an MR HBase table
>>>>> loader) on latest build of 0.92.0 (r1232715) I get an 'old server' warning
>>>>> (I've seen this before and it's always been non-fatal) then an out of
>>>>> memory
>>>>> exception then job hangs:
>>>>>
>>>>>
>>>>>
>>>>> [sshexec] 12/01/20 16:56:48 WARN zookeeper.ClientCnxnSocket: Connected to
>>>>> an
>>>>> old server; r-o mode will be unavailable
>>>>>
>>>>> [sshexec] 12/01/20 16:56:48 INFO zookeeper.ClientCnxn: Session
>>>>> establishment complete on server namenode/10.0.0.235:2181, sessionid =
>>>>> 0x34cda4e5d000e5, negotiated timeout = 40000
>>>>>
>>>>> [sshexec] 12/01/20 16:56:49 WARN ipc.HBaseClient: Unexpected exception
>>>>> receiving call responses
>>>>>
>>>>> [sshexec] java.lang.OutOfMemoryError: Java heap space
>>>>>
>>>>> [sshexec] at java.lang.reflect.Array.newArray(Native Method)
>>>>>
>>>>> [sshexec] at java.lang.reflect.Array.newInstance(Array.java:52)
>>>>>
>>>>> [sshexec] at
>>>>>
>>>>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritabl
>>>>> e.java:542)
>>>>>
>>>>> [sshexec] at
>>>>>
>>>>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritabl
>>>>> e.java:289)
>>>>>
>>>>> [sshexec] at
>>>>>
>>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
>>>>> nt.java:593)
>>>>>
>>>>> [sshexec] at
>>>>>
>>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
>>>>>
>>>>>
>>>>>
>>>>> Hbase shell seems to work (I can list and scan my tables).
>>>>>
>>>>>
>>>>>
>>>>> If I svn roll back to 12 Jan 0.92 and rebuild my code works.
>>>>>
>>>>>
>>>>>
>>>>> Tried setting export HBASE_HEAPSIZE=1500 but got same error.
>>>>>
>>>>>
>>>>>
>>>>> Nothing significant in logs.
>>>>>
>>>>>
>>>>>
>>>>> [Note to Ted Yu: I need to fix this so I can carry on testing on
>>>>> Aggregation
>>>>> Protocol]
>>>>>
>>>>>
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Royston
>>>>>
>>>>>
>>>
>

Re: Hbase out of memory error

Royston Sellman 2012-01-21, 17:36

Hmmm, when I look at the source that I *think* I am building HBase from I can see Benoit's patch is there. But when I look in the hbase-0.92.0-sources.jar that gets built at the same time the patch is not there. I better check my build process...

Cheers,
Royston

On 21 Jan 2012, at 17:13, [EMAIL PROTECTED] wrote:

> Benoit's patches are already in 0.92
>
> Thanks
>
>
> On Jan 21, 2012, at 9:11 AM, Royston Sellman <[EMAIL PROTECTED]> wrote:
>
>> So should I try applying Benoit Sigoure's patch for HBASE-5204? Will this patch be in the 0.92 branch soon?
>>
>> Cheers,
>> Royston
>>
>>
>>
>> On 21 Jan 2012, at 16:58, [EMAIL PROTECTED] wrote:
>>
>>> That is the correct branch.
>>>
>>> Thanks
>>>
>>>
>>>
>>> On Jan 21, 2012, at 8:50 AM, Royston Sellman <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi Ted,
>>>>
>>>> Yes, I am compiling with the same HBase jars. I wasn't aware of HBASE-5204, thanks, it sounds possible this is my problem. Can you think of anything else I should check?
>>>>
>>>> Just to make sure: I am checking out the code from svn.apache.org/repos/asf/hbase/branches/0.92 Is this the correct branch?
>>>>
>>>> Thanks,
>>>> Royston
>>>>
>>>>
>>>> On 20 Jan 2012, at 18:45, Ted Yu wrote:
>>>>
>>>>> Royston:
>>>>> I guess you have seen HBASE-5204. In particular:
>>>>>>> when a 0.92 server fails to deserialize a 0.90-style RPC, it attempts to
>>>>> allocate a large buffer because it doesn't read fields of 0.90-style RPCs
>>>>> properly.
>>>>>
>>>>> Were your client code compiled with the same version of HBase as what was
>>>>> running on your cluster ?
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Fri, Jan 20, 2012 at 9:20 AM, Royston Sellman <
>>>>> [EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Trying to run my code (a test of Aggregation Protocol and an MR HBase table
>>>>>> loader) on latest build of 0.92.0 (r1232715) I get an 'old server' warning
>>>>>> (I've seen this before and it's always been non-fatal) then an out of
>>>>>> memory
>>>>>> exception then job hangs:
>>>>>>
>>>>>>
>>>>>>
>>>>>> [sshexec] 12/01/20 16:56:48 WARN zookeeper.ClientCnxnSocket: Connected to
>>>>>> an
>>>>>> old server; r-o mode will be unavailable
>>>>>>
>>>>>> [sshexec] 12/01/20 16:56:48 INFO zookeeper.ClientCnxn: Session
>>>>>> establishment complete on server namenode/10.0.0.235:2181, sessionid >>>>>> 0x34cda4e5d000e5, negotiated timeout = 40000
>>>>>>
>>>>>> [sshexec] 12/01/20 16:56:49 WARN ipc.HBaseClient: Unexpected exception
>>>>>> receiving call responses
>>>>>>
>>>>>> [sshexec] java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>> [sshexec] at java.lang.reflect.Array.newArray(Native Method)
>>>>>>
>>>>>> [sshexec] at java.lang.reflect.Array.newInstance(Array.java:52)
>>>>>>
>>>>>> [sshexec] at
>>>>>>
>>>>>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritabl
>>>>>> e.java:542)
>>>>>>
>>>>>> [sshexec] at
>>>>>>
>>>>>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritabl
>>>>>> e.java:289)
>>>>>>
>>>>>> [sshexec] at
>>>>>>
>>>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
>>>>>> nt.java:593)
>>>>>>
>>>>>> [sshexec] at
>>>>>>
>>>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hbase shell seems to work (I can list and scan my tables).
>>>>>>
>>>>>>
>>>>>>
>>>>>> If I svn roll back to 12 Jan 0.92 and rebuild my code works.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Tried setting export HBASE_HEAPSIZE=1500 but got same error.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Nothing significant in logs.
>>>>>>
>>>>>>
>>>>>>
>>>>>> [Note to Ted Yu: I need to fix this so I can carry on testing on
>>>>>> Aggregation
>>>>>> Protocol]
>>>>>>
>>>>>>
>>>>>>
>>>>>> Best Regards,
>>>>>>
>>>>>> Royston
>>>>>>
>>>>>>
>>>>
>>

RE: Hbase out of memory error

Royston Sellman 2012-01-23, 13:50

Hi Ted,

Finally rebuilt branch/0.92 and applied your patch and rebuilt my code.
Using AggregationClient.sum() on my test table I get the correct result.
Just swapping to AggregationClient.median() I get the following error:

  [sshexec] org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=10, exceptions:
  [sshexec] Mon Jan 23 13:44:12 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:13 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:14 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:15 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:17 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:19 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:23 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:27 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:35 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec] Mon Jan 23 13:44:51 GMT 2012,
org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
java.lang.NullPointerException
  [sshexec]
  [sshexec] Result = -1
  [sshexec] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
getRegionServerWithRetries(HConnectionManager.java:1345)
  [sshexec] at
org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:
1203)
  [sshexec] at
org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1
126)
  [sshexec] at
org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:627)
  [sshexec] at
org.apache.hadoop.hbase.client.coprocessor.AggregationClient.median(Aggregat
ionClient.java:469)
  [sshexec] at
uk.org.cse.aggregation.EDRPAggregator.testSumWithValidRange(EDRPAggregator.j
ava:55)
  [sshexec] at
uk.org.cse.aggregation.EDRPAggregator.main(EDRPAggregator.java:85)
  [sshexec] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
  [sshexec] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
  [sshexec] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
  [sshexec] at java.lang.reflect.Method.invoke(Method.java:597)
  [sshexec] at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

Something wrong with Scan setup?

Cheers,
Royston

-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
Sent: 21 January 2012 17:14
To: [EMAIL PROTECTED]
Subject: Re: Hbase out of memory error

Benoit's patches are already in 0.92

Thanks
On Jan 21, 2012, at 9:11 AM, Royston Sellman
<[EMAIL PROTECTED]> wrote:

> So should I try applying Benoit Sigoure's patch for HBASE-5204? Will this
patch be in the 0.92 branch soon?
>
> Cheers,
> Royston
>
>
>
> On 21 Jan 2012, at 16:58, [EMAIL PROTECTED] wrote:
>
>> That is the correct branch.
>>
>> Thanks
>>
>>
>>
>> On Jan 21, 2012, at 8:50 AM, Royston Sellman
<[EMAIL PROTECTED]> wrote:
>>
>>> Hi Ted,
>>>
>>> Yes, I am compiling with the same HBase jars. I wasn't aware of
HBASE-5204, thanks, it sounds possible this is my problem. Can you think of
anything else I should check?
>>>
>>> Just to make sure: I am checking out the code from
svn.apache.org/repos/asf/hbase/branches/0.92 Is this the correct branch?
>>>
>>> Thanks,
>>> Royston
>>>
>>>
>>> On 20 Jan 2012, at 18:45, Ted Yu wrote:
>>>
>>>> Royston:
>>>> I guess you have seen HBASE-5204. In particular: