Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> NullPointerException while loading large amount of new rows into HBase, exception is thrown when trying to obtain lock for RowKey

Copy link to this message
NullPointerException while loading large amount of new rows into HBase, exception is thrown when trying to obtain lock for RowKey

When our customers(using TSDB) loads large amount of data into HBase, we saw many NullPointerException in the RS logs as below. I checked the source code, it seems when trying to obtain the lock for a rowKey, if the entry for that row already exists and the "waitfoBlock" is false (therefore it won't retry, but just return a NULL value). I can see in doMiniBatchMutation(), the waitForBlock is set to be false. (most other places "waitForBlock" is always set to true).

This exception is thrown from function lockRow(), which has been deprecated. I am not sure why it is deprecated, and what is used to replace it. Is this normal? which implies the HBase should not throw this misleading error message to log.  Or should the client call some other API?


2013-05-14 12:45:30,911 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock -3430274391270203797 explicitly acquired by client
2013-05-14 12:45:30,911 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":29783,"call":"lockRow([B@339a6a5c, [B@5ebcd87b), rpc version=1, client version=29, methodsFingerPrint=0","client":"","starttimems":1368560701128,"queuetimems":847,"class":"HRegionServer","responsesize":0,"method":"lockRow"}
2013-05-14 12:46:00,911 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Error obtaining row lock (fsOk: true)
                at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.addRowLock(HRegionServer.java:2346)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.lockRow(HRegionServer.java:2332)
                at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:384)
                at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call get([B@2166c821, {"timeRange":[0,9223372036854775807],"totalColumns":1,"cacheBlocks":true,"families":{"id":["tagv"]},"maxVersions":1,"row":"slcsn-s00314.slc.ebay.com"}), rpc version=1, client version=29, methodsFingerPrint=0 from output error
2013-05-14 12:46:02,514 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020 caught: java.io.IOException: Connection reset by peer
                at sun.nio.ch.FileDispatcher.write0(Native Method)
                at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
ramkrishna vasudevan 2013-05-16, 17:16
ramkrishna vasudevan 2013-05-16, 17:41
Tianying Chang 2013-05-16, 18:35
Tianying Chang 2013-05-16, 18:41
ramkrishna vasudevan 2013-05-17, 03:49
Tianying Chang 2013-05-20, 17:32