-
The Exceptions When Concurrently Writing and ReadingBing Li 2013-02-05, 18:39
Dear all,
To raise the performance of writing data into HBase, the
"synchronized" is removed from the writing method.
But after "synchronized" is removed from the method of writing, I get
the following exceptions when reading. Before the removal, no such
exceptions.
Could you help me how to solve it?
Thanks so much!
Best wishes,
Bing
Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
WARNING: Unexpected exception receiving call responses
java.lang.NullPointerException
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.client.ScannerCallable close
WARNING: Ignore, probably already closed
java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
failed on local exception: java.io.IOException: Unexpected exception
receiving call responses
at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy6.close(Unknown Source)
at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
at com.greatfree.hbase.rank.NodeRankRetriever.LoadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
at com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Caused by:
java.io.IOException: Unexpected exception receiving call responses
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:509)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
The writing method is as follows.
// The "synchronized" is removed to raise the performance.
// public synchronized void AddNodeViewGroupNodeRanks(String
hostNodeKey, String groupKey, int timingScale, Map<String, Double>
groupNodeRankMap)
public void AddNodeViewGroupNodeRanks(String hostNodeKey, String
groupKey, int timingScale, Map<String, Double> groupNodeRankMap)
{
List<Put> puts = new ArrayList<Put>();
Put hostNodeKeyPut;
Put groupKeyPut;
Put timingScalePut;
Put nodeKeyPut;
Put rankPut;
byte[] groupNodeRankRowKey;
for (Map.Entry<String, Double> nodeRankEntry :
groupNodeRankMap.entrySet())
{
groupNodeRankRowKey = Bytes.toBytes(...);
hostNodeKeyPut = new Put(groupNodeRankRowKey);
hostNodeKeyPut.add(...);
puts.add(hostNodeKeyPut);
......
rankPut = new Put(groupNodeRankRowKey);
rankPut.add(...);
puts.add(rankPut);
}
try
{
this.rankTable.put(puts);
}
catch (IOException e)
{
e.printStackTrace();
}
}
The reading method that causes the exceptions is as follows.
public Set<String> LoadNodeGroupNodeRankRowKeys(String
hostNodeKey, String groupKey, int timingScale)
{
List<Filter> nodeGroupFilterList = new ArrayList<Filter>();
SingleColumnValueFilter hostNodeKeyFilter = new
SingleColumnValueFilter(...);
hostNodeKeyFilter.setFilterIfMissing(true);
nodeGroupFilterList.add(hostNodeKeyFilter);
......
FilterList nodeGroupFilter = new
FilterList(nodeGroupFilterList);
Scan scan = new Scan();
scan.setFilter(nodeGroupFilter);
scan.setCaching(Parameters.CACHING_SIZE);
scan.setBatch(Parameters.BATCHING_SIZE);
Set<String> rowKeySet = Sets.newHashSet();
try
{
ResultScanner scanner = this.rankTable.getScanner(scan);
// EXCEPTIONS are raised at the following line.
for (Result result : scanner)
{
for (KeyValue kv : result.raw())
{
rowKeySet.add(Bytes.toString(kv.getRow()));
break;
}
}
scanner.close();
}
catch (IOException e)
{
e.pri
To raise the performance of writing data into HBase, the
"synchronized" is removed from the writing method.
But after "synchronized" is removed from the method of writing, I get
the following exceptions when reading. Before the removal, no such
exceptions.
Could you help me how to solve it?
Thanks so much!
Best wishes,
Bing
Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
WARNING: Unexpected exception receiving call responses
java.lang.NullPointerException
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.client.ScannerCallable close
WARNING: Ignore, probably already closed
java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
failed on local exception: java.io.IOException: Unexpected exception
receiving call responses
at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy6.close(Unknown Source)
at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
at com.greatfree.hbase.rank.NodeRankRetriever.LoadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
at com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Caused by:
java.io.IOException: Unexpected exception receiving call responses
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:509)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
The writing method is as follows.
// The "synchronized" is removed to raise the performance.
// public synchronized void AddNodeViewGroupNodeRanks(String
hostNodeKey, String groupKey, int timingScale, Map<String, Double>
groupNodeRankMap)
public void AddNodeViewGroupNodeRanks(String hostNodeKey, String
groupKey, int timingScale, Map<String, Double> groupNodeRankMap)
{
List<Put> puts = new ArrayList<Put>();
Put hostNodeKeyPut;
Put groupKeyPut;
Put timingScalePut;
Put nodeKeyPut;
Put rankPut;
byte[] groupNodeRankRowKey;
for (Map.Entry<String, Double> nodeRankEntry :
groupNodeRankMap.entrySet())
{
groupNodeRankRowKey = Bytes.toBytes(...);
hostNodeKeyPut = new Put(groupNodeRankRowKey);
hostNodeKeyPut.add(...);
puts.add(hostNodeKeyPut);
......
rankPut = new Put(groupNodeRankRowKey);
rankPut.add(...);
puts.add(rankPut);
}
try
{
this.rankTable.put(puts);
}
catch (IOException e)
{
e.printStackTrace();
}
}
The reading method that causes the exceptions is as follows.
public Set<String> LoadNodeGroupNodeRankRowKeys(String
hostNodeKey, String groupKey, int timingScale)
{
List<Filter> nodeGroupFilterList = new ArrayList<Filter>();
SingleColumnValueFilter hostNodeKeyFilter = new
SingleColumnValueFilter(...);
hostNodeKeyFilter.setFilterIfMissing(true);
nodeGroupFilterList.add(hostNodeKeyFilter);
......
FilterList nodeGroupFilter = new
FilterList(nodeGroupFilterList);
Scan scan = new Scan();
scan.setFilter(nodeGroupFilter);
scan.setCaching(Parameters.CACHING_SIZE);
scan.setBatch(Parameters.BATCHING_SIZE);
Set<String> rowKeySet = Sets.newHashSet();
try
{
ResultScanner scanner = this.rankTable.getScanner(scan);
// EXCEPTIONS are raised at the following line.
for (Result result : scanner)
{
for (KeyValue kv : result.raw())
{
rowKeySet.add(Bytes.toString(kv.getRow()));
break;
}
}
scanner.close();
}
catch (IOException e)
{
e.pri
+
Ted Yu 2013-02-05, 18:45
+
Bing Li 2013-02-05, 18:46
+
Ted Yu 2013-02-05, 18:49
+
Bing Li 2013-02-05, 19:00
+
Bing Li 2013-02-05, 19:20
