Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> NullPointerException while loading large amount of new rows into HBase, exception is thrown when trying to obtain lock for RowKey


Copy link to this message
-
NullPointerException while loading large amount of new rows into HBase, exception is thrown when trying to obtain lock for RowKey
Hi,

When our customers(using TSDB) loads large amount of data into HBase, we saw many NullPointerException in the RS logs as below. I checked the source code, it seems when trying to obtain the lock for a rowKey, if the entry for that row already exists and the "waitfoBlock" is false (therefore it won't retry, but just return a NULL value). I can see in doMiniBatchMutation(), the waitForBlock is set to be false. (most other places "waitForBlock" is always set to true).

This exception is thrown from function lockRow(), which has been deprecated. I am not sure why it is deprecated, and what is used to replace it. Is this normal? which implies the HBase should not throw this misleading error message to log.  Or should the client call some other API?

Thanks
Tian-Ying

2013-05-14 12:45:30,911 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock -3430274391270203797 explicitly acquired by client
2013-05-14 12:45:30,911 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":29783,"call":"lockRow([B@339a6a5c, [B@5ebcd87b), rpc version=1, client version=29, methodsFingerPrint=0","client":"10.53.106.37:58892","starttimems":1368560701128,"queuetimems":847,"class":"HRegionServer","responsesize":0,"method":"lockRow"}
2013-05-14 12:46:00,911 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Error obtaining row lock (fsOk: true)
java.lang.NullPointerException
                at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.addRowLock(HRegionServer.java:2346)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.lockRow(HRegionServer.java:2332)
                at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:384)
                at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@2166c821, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"id":["tagv"]},"maxVersions":1,"row":"slcsn-s00314.slc.ebay.com"}), rpc version=1, client version=29, methodsFingerPrint=0 from 10.53.106.37:58892: output error
2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020 caught: java.io.IOException: Connection reset by peer
                at sun.nio.ch.FileDispatcher.write0(Native Method)
                at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
+
ramkrishna vasudevan 2013-05-16, 17:16
+
ramkrishna vasudevan 2013-05-16, 17:41
+
Tianying Chang 2013-05-16, 18:35
+
Tianying Chang 2013-05-16, 18:41
+
ramkrishna vasudevan 2013-05-17, 03:49
+
Tianying Chang 2013-05-20, 17:32
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB