Royston:
The exception came from this line:
    ResultScanner scanner = table.getScanner(scan2);
Can you help me review the logic starting with:
    // scan the region with median and find it
    Scan scan2 = new Scan(scan);
You can log the String form of scan and scan2 before the table.getScanner()
call.

I think the NPE below reveals that startRow is null (median is in first
region).
If that is the case, the following should help:
    if (startRow != null) scan2.setStartRow(startRow);

Thanks

On Mon, Jan 23, 2012 at 5:50 AM, Royston Sellman <
[EMAIL PROTECTED]> wrote:

> Hi Ted,
>
> Finally rebuilt branch/0.92 and applied your patch and rebuilt my code.
> Using AggregationClient.sum() on my test table I get the correct result.
> Just swapping to AggregationClient.median() I get the following error:
>
>  [sshexec] org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after attempts=10, exceptions:
>  [sshexec] Mon Jan 23 13:44:12 GMT 2012,
> org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
> java.lang.NullPointerException
>  [sshexec] Mon Jan 23 13:44:13 GMT 2012,
> org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
> java.lang.NullPointerException
>  [sshexec] Mon Jan 23 13:44:14 GMT 2012,
> org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
> java.lang.NullPointerException
>  [sshexec] Mon Jan 23 13:44:15 GMT 2012,
> org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
> java.lang.NullPointerException
>  [sshexec] Mon Jan 23 13:44:17 GMT 2012,
> org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
> java.lang.NullPointerException
>  [sshexec] Mon Jan 23 13:44:19 GMT 2012,
> org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
> java.lang.NullPointerException
>  [sshexec] Mon Jan 23 13:44:23 GMT 2012,
> org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
> java.lang.NullPointerException
>  [sshexec] Mon Jan 23 13:44:27 GMT 2012,
> org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
> java.lang.NullPointerException
>  [sshexec] Mon Jan 23 13:44:35 GMT 2012,
> org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
> java.lang.NullPointerException
>  [sshexec] Mon Jan 23 13:44:51 GMT 2012,
> org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
> java.lang.NullPointerException
>  [sshexec]
>  [sshexec] Result = -1
>  [sshexec]     at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> getRegionServerWithRetries(HConnectionManager.java:1345)
>  [sshexec]     at
>
> org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:
> 1203)
>  [sshexec]     at
>
> org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1
> 126)
>  [sshexec]     at
> org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:627)
>  [sshexec]     at
>
> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.median(Aggregat
> ionClient.java:469)
>  [sshexec]     at
>
> uk.org.cse.aggregation.EDRPAggregator.testSumWithValidRange(EDRPAggregator.j
> ava:55)
>  [sshexec]     at
> uk.org.cse.aggregation.EDRPAggregator.main(EDRPAggregator.java:85)
>  [sshexec]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>  [sshexec]     at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> )
>  [sshexec]     at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>  [sshexec]     at java.lang.reflect.Method.invoke(Method.java:597)
>  [sshexec]     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> Something wrong with Scan setup?
>
> Cheers,
> Royston
>
>
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
> Sent: 21 January 2012 17:14
> To: [EMAIL PROTECTED]
> Subject: Re: Hbase out of memory error
>
> Benoit's patches are already in 0.92
>
> Thanks
>
>
> On Jan 21, 2012, at 9:11 AM, Royston Sellman
> <[EMAIL PROTECTED]> wrote:
>
> > So should I try applying Benoit Sigoure's patch for HBASE-5204? Will this

Hi Ted,

Following from what you have said, we have edited AggregateClient.java with the following modification to the median() method:

...

    // scan the region with median and find it
    Scan scan2 = new Scan(scan);
    // inherit stop row from method parameter
    if (startRow != null)
      scan2.setStartRow(startRow);
    HTable table = new HTable(conf, tableName);
    int cacheSize = scan2.getCaching();
    if (!scan2.getCacheBlocks() || scan2.getCaching() < 2) {
      scan2.setCacheBlocks(true);
      cacheSize = 5;
      scan2.setCaching(cacheSize);
    }

//    log.debug("scan:\t" + scan);
//    log.debug("scan2:\t" + scan2);

    ResultScanner scanner = table.getScanner(scan2);

...

The addition of "if (startRow != null) scan2.setStartRow(startRow);" has fixed our NPE/Memory issue and we now see a sensible result successfully returned.

However, in the process of debugging this, I added some 'log.debug("scan:\t" + scan);' statements as you can see in the snippet above. These lines (when uncommented) were causing the following exception:

[sshexec] java.lang.NoSuchMethodError: org.codehaus.jackson.map.ObjectMapper.writeValueAsString(Ljava/lang/Object;)Ljava/lang/String;
  [sshexec]
  [sshexec] at org.apache.hadoop.hbase.client.Operation.toJSON(Operation.java:67)
  [sshexec]
  [sshexec] at org.apache.hadoop.hbase.client.Operation.toString(Operation.java:93)
  [sshexec]
  [sshexec] at org.apache.hadoop.hbase.client.Operation.toString(Operation.java:107)
  [sshexec]
  [sshexec] at java.lang.String.valueOf(String.java:2826)
  [sshexec] at java.lang.StringBuilder.append(StringBuilder.java:115)
  [sshexec] at org.apache.hadoop.hbase.client.coprocessor.AggregationClient.median(AggregationClient.java:473)
  [sshexec] at uk.org.cse.aggregation.EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:55)
  [sshexec] at uk.org.cse.aggregation.EDRPAggregator.main(EDRPAggregator.java:85)
  [sshexec] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  [sshexec] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  [sshexec] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  [sshexec] at java.lang.reflect.Method.invoke(Method.java:597)
  [sshexec] at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

Do you have any idea why we got this error above? It seems the scanner toString() method requires the Jackson JSON processor, which is failing for some reason...

Also, please can you tell me where log.debug() output can be found? (e.g. $HBASE_HOME/logs/xxx-region-....log?)

Please find a patch attached with the fixed median method for HBase 0.92 branch. Note that I have removed the log.debug() statements.

Thanks,
Tom

-----Original Message-----
From: Ted Yu [mailto:[EMAIL PROTECTED]]
Sent: 23 January 2012 15:24
To: [EMAIL PROTECTED]
Subject: median through AggregationProtocol

Royston:
The exception came from this line:
    ResultScanner scanner = table.getScanner(scan2); Can you help me review the logic starting with:
    // scan the region with median and find it
    Scan scan2 = new Scan(scan);
You can log the String form of scan and scan2 before the table.getScanner() call.

I think the NPE below reveals that startRow is null (median is in first region).
If that is the case, the following should help:
    if (startRow != null) scan2.setStartRow(startRow);

Thanks

On Mon, Jan 23, 2012 at 5:50 AM, Royston Sellman < [EMAIL PROTECTED]> wrote:

> Hi Ted,
>
> Finally rebuilt branch/0.92 and applied your patch and rebuilt my code.
> Using AggregationClient.sum() on my test table I get the correct result.
> Just swapping to AggregationClient.median() I get the following error:
>
>  [sshexec] org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Failed after attempts=10, exceptions:
>  [sshexec] Mon Jan 23 13:44:12 GMT 2012,
> org.apache.hadoop.hbase.client.ScannerCallable@219ba640,
> java.lang.NullPointerException

Thanks Tom for the investigation.
I will apply the null check in an addendum to HBASE-5139.

Operation.toJSON() does bring in Jackson JSON processor
Please confirm that the jackson jars are in your CLASSPATH:
$ ls lib/*jackson*
lib/jackson-core-asl-1.5.5.jar  lib/jackson-jaxrs-1.5.5.jar
lib/jackson-mapper-asl-1.5.5.jar  lib/jackson-xc-1.5.5.jar

Looks like you haven't encountered the need to print contents of Scan so
far.
If you feel the dependency on jackson is burdensome, please file a JIRA so
that we can relax this dependency.

Cheers

On Mon, Jan 23, 2012 at 8:35 AM, Tom Wilcox <[EMAIL PROTECTED]> wrote:

> Hi Ted,
>
> Following from what you have said, we have edited AggregateClient.java
> with the following modification to the median() method:
>
> ...
>
>    // scan the region with median and find it
>    Scan scan2 = new Scan(scan);
>     // inherit stop row from method parameter
>     if (startRow != null)
>      scan2.setStartRow(startRow);
>     HTable table = new HTable(conf, tableName);
>    int cacheSize = scan2.getCaching();
>    if (!scan2.getCacheBlocks() || scan2.getCaching() < 2) {
>      scan2.setCacheBlocks(true);
>      cacheSize = 5;
>      scan2.setCaching(cacheSize);
>    }
>
> //    log.debug("scan:\t" + scan);
> //    log.debug("scan2:\t" + scan2);
>
>    ResultScanner scanner = table.getScanner(scan2);
>
> ...
>
> The addition of "if (startRow != null) scan2.setStartRow(startRow);" has
> fixed our NPE/Memory issue and we now see a sensible result successfully
> returned.
>
> However, in the process of debugging this, I added some
> 'log.debug("scan:\t" + scan);' statements as you can see in the snippet
> above. These lines (when uncommented) were causing the following exception:
>
> [sshexec] java.lang.NoSuchMethodError:
> org.codehaus.jackson.map.ObjectMapper.writeValueAsString(Ljava/lang/Object;)Ljava/lang/String;
>  [sshexec]
>  [sshexec]     at
> org.apache.hadoop.hbase.client.Operation.toJSON(Operation.java:67)
>  [sshexec]
>  [sshexec]     at
> org.apache.hadoop.hbase.client.Operation.toString(Operation.java:93)
>  [sshexec]
>  [sshexec]     at
> org.apache.hadoop.hbase.client.Operation.toString(Operation.java:107)
>  [sshexec]
>  [sshexec]     at java.lang.String.valueOf(String.java:2826)
>  [sshexec]     at java.lang.StringBuilder.append(StringBuilder.java:115)
>  [sshexec]     at
> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.median(AggregationClient.java:473)
>  [sshexec]     at
> uk.org.cse.aggregation.EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:55)
>   [sshexec]     at
> uk.org.cse.aggregation.EDRPAggregator.main(EDRPAggregator.java:85)
>  [sshexec]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>  [sshexec]     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   [sshexec]     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   [sshexec]     at java.lang.reflect.Method.invoke(Method.java:597)
>  [sshexec]     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> Do you have any idea why we got this error above? It seems the scanner
> toString() method requires the Jackson JSON processor, which is failing for
> some reason...
>
> Also, please can you tell me where log.debug() output can be found? (e.g.
> $HBASE_HOME/logs/xxx-region-....log?)
>
> Please find a patch attached with the fixed median method for HBase 0.92
> branch. Note that I have removed the log.debug() statements.
>
> Thanks,
> Tom
>
> -----Original Message-----
> From: Ted Yu [mailto:[EMAIL PROTECTED]]
> Sent: 23 January 2012 15:24
> To: [EMAIL PROTECTED]
> Subject: median through AggregationProtocol
>
> Royston:
> The exception came from this line:
>    ResultScanner scanner = table.getScanner(scan2); Can you help me review
> the logic starting with:
>    // scan the region with median and find it
>    Scan scan2 = new Scan(scan);
> You can log the String form of scan and scan2 before the

Royston / Tom:
I would encourage you to explore other aggregations where
AggregationProtocol is of help.

Feel free to discuss any limitation in current implementation, propose
suggestions, etc.

Thanks

On Mon, Jan 23, 2012 at 9:03 AM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Thanks Tom for the investigation.
> I will apply the null check in an addendum to HBASE-5139.
>
> Operation.toJSON() does bring in Jackson JSON processor
> Please confirm that the jackson jars are in your CLASSPATH:
> $ ls lib/*jackson*
> lib/jackson-core-asl-1.5.5.jar  lib/jackson-jaxrs-1.5.5.jar
> lib/jackson-mapper-asl-1.5.5.jar  lib/jackson-xc-1.5.5.jar
>
> Looks like you haven't encountered the need to print contents of Scan so
> far.
> If you feel the dependency on jackson is burdensome, please file a JIRA so
> that we can relax this dependency.
>
> Cheers
>
>
> On Mon, Jan 23, 2012 at 8:35 AM, Tom Wilcox <[EMAIL PROTECTED]> wrote:
>
>> Hi Ted,
>>
>> Following from what you have said, we have edited AggregateClient.java
>> with the following modification to the median() method:
>>
>> ...
>>
>>    // scan the region with median and find it
>>    Scan scan2 = new Scan(scan);
>>     // inherit stop row from method parameter
>>     if (startRow != null)
>>      scan2.setStartRow(startRow);
>>     HTable table = new HTable(conf, tableName);
>>    int cacheSize = scan2.getCaching();
>>    if (!scan2.getCacheBlocks() || scan2.getCaching() < 2) {
>>      scan2.setCacheBlocks(true);
>>      cacheSize = 5;
>>      scan2.setCaching(cacheSize);
>>    }
>>
>> //    log.debug("scan:\t" + scan);
>> //    log.debug("scan2:\t" + scan2);
>>
>>    ResultScanner scanner = table.getScanner(scan2);
>>
>> ...
>>
>> The addition of "if (startRow != null) scan2.setStartRow(startRow);" has
>> fixed our NPE/Memory issue and we now see a sensible result successfully
>> returned.
>>
>> However, in the process of debugging this, I added some
>> 'log.debug("scan:\t" + scan);' statements as you can see in the snippet
>> above. These lines (when uncommented) were causing the following exception:
>>
>> [sshexec] java.lang.NoSuchMethodError:
>> org.codehaus.jackson.map.ObjectMapper.writeValueAsString(Ljava/lang/Object;)Ljava/lang/String;
>>  [sshexec]
>>  [sshexec]     at
>> org.apache.hadoop.hbase.client.Operation.toJSON(Operation.java:67)
>>  [sshexec]
>>  [sshexec]     at
>> org.apache.hadoop.hbase.client.Operation.toString(Operation.java:93)
>>  [sshexec]
>>  [sshexec]     at
>> org.apache.hadoop.hbase.client.Operation.toString(Operation.java:107)
>>  [sshexec]
>>  [sshexec]     at java.lang.String.valueOf(String.java:2826)
>>  [sshexec]     at java.lang.StringBuilder.append(StringBuilder.java:115)
>>  [sshexec]     at
>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.median(AggregationClient.java:473)
>>  [sshexec]     at
>> uk.org.cse.aggregation.EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:55)
>>   [sshexec]     at
>> uk.org.cse.aggregation.EDRPAggregator.main(EDRPAggregator.java:85)
>>  [sshexec]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>>  [sshexec]     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>   [sshexec]     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>   [sshexec]     at java.lang.reflect.Method.invoke(Method.java:597)
>>  [sshexec]     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>
>> Do you have any idea why we got this error above? It seems the scanner
>> toString() method requires the Jackson JSON processor, which is failing for
>> some reason...
>>
>> Also, please can you tell me where log.debug() output can be found? (e.g.
>> $HBASE_HOME/logs/xxx-region-....log?)
>>
>> Please find a patch attached with the fixed median method for HBase 0.92
>> branch. Note that I have removed the log.debug() statements.
>>
>> Thanks,
>> Tom
>>
>> -----Original Message-----
>> From: Ted Yu [mailto:[EMAIL PROTECTED]]
>> Sent: 23 January 2012 15:24