Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - openTSDB   lose large amount of data when the client are writing


Copy link to this message
-
Re: openTSDB lose large amount of data when the client are writing
Stack 2013-09-19, 18:02
On Thu, Sep 19, 2013 at 10:09 AM, Tianying Chang <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I have a customer who use openTSDB. Recently we found that only less than
> 10% data are written, rest are are lost. By checking the RS log, there are
> many row lock related issues, like below. It seems large amount of write to
> tsdb that need row lock caused the problem. Anyone else see similar
> problem?  Is it a bug of openTSDB? Or it is due to HBase exposed a
> vulnerable API?
>
> org.apache.hadoop.hbase.UnknownRowLockException: Invalid row lock
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getLockFromId(HRegionServer.java:2732)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2071)
>         at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
> 13/09/18 12:08:30 ERROR regionserver.HRegionServer:
> org.apache.hadoop.hbase.UnknownRowLockException: -6180307918863136448
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.unlockRow(HRegionServer.java:2765)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
>
>
> Thanks
> Tian-Ying
>
Local filesystem or hdfs?
St.Ack