Hi there

I am getting an UnknownRowLockException when adding locks to the increment() function below. The full stack trace is at the end of this message.

There are a few other places I am acquiring locks, but I only ever acquire a single lock for one piece of code and the rest go through fine. It's only when I enable the locks in this function that I get the exception. It's fairly simple, and I can't find anything obviously wrong with it so my best guess is there's some quirk with hbase locks that I'm unaware of. Any ideas?

One thing to note is that I am calling increment() a couple times consecutively for the same row.

Marco

  public static void increment(String tableName, String key, String family,
                         String qualifier, int value, boolean useShort,
                         Configuration config)
      throws IOException{
    HTable table = new HTable(config,
        config.get("app", "geomon-test") + "." + tableName);
    Get get = new Get(key.getBytes());
    get.addColumn(family.getBytes(),
        qualifier.getBytes());
    int before = 0;
    RowLock lock = table.lockRow(key.getBytes());
    try {
      Result result = table.get(get);
      if (!result.isEmpty()) {
        if (useShort) {
          before = (int) Bytes.toShort(result.getValue(family.getBytes(),
              qualifier.getBytes()));
        } else {
          before = Bytes.toInt(result.getValue(family.getBytes(),
              qualifier.getBytes()));
        }
      }
      if (useShort) {
        value = Math.min(before + value, Short.MAX_VALUE);
      } else {
        value = (int) Math.min((long) before + value, Integer.MAX_VALUE);
      }

      Put put = new Put(key.getBytes());
      put.add(family.getBytes(),
          qualifier.getBytes(),
          Bytes.toBytes(useShort ? (short) value : value));
      table.put(put);
    } catch (Exception e) {
      System.err.println(e.getMessage());
    } finally {
      table.unlockRow(lock);
    }
  }
12/07/24 14:20:58 INFO mapred.JobClient: Task Id : attempt_201207241320_0002_r_000000_2, Status : FAILEDorg.apache.hadoop.hbase.UnknownRowLockException: org.apache.hadoop.hbase.UnknownRowLockException: -3110061788478328763
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)        at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
        at org.apache.hadoop.hbase.client.ServerCallable.translateException(ServerCallable.java:228)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:166)
        at org.apache.hadoop.hbase.client.HTable.unlockRow(HTable.java:1031)
        at Util.increment(Util.java:170)
        at Aggregate$EventReducer.processUserEvent(Aggregate.java:161)
        at Aggregate$EventReducer.processUser(Aggregate.java:119)
        at Aggregate$EventReducer.reduce(Aggregate.java:189)
        at Aggregate$EventReducer.reduce(Aggregate.java:78)
        at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:416)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.UnknownRowLockException: -3110061788478328763
        at org.apache.hadoop.hbase.regionserver.HRegionServer.unlockRow(HRegionServer.java:2633)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)

        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
        at $Proxy3.unlockRow(Unknown Source)
        at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1033)
        at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1031)
        at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:163)
        ... 14 more

Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

Row locks don't move with regions or splits... are such things
happening frequently?

Also, any reason to not use HBase's own increment method that's much
more efficient?

http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#incrementColumnValue(byte[],
byte[], byte[], long)

J-D

On Tue, Jul 24, 2012 at 2:53 PM, Marco Gallotta <[EMAIL PROTECTED]> wrote:
> Hi there
>
> I am getting an UnknownRowLockException when adding locks to the increment() function below. The full stack trace is at the end of this message.
>
> There are a few other places I am acquiring locks, but I only ever acquire a single lock for one piece of code and the rest go through fine. It's only when I enable the locks in this function that I get the exception. It's fairly simple, and I can't find anything obviously wrong with it so my best guess is there's some quirk with hbase locks that I'm unaware of. Any ideas?
>
> One thing to note is that I am calling increment() a couple times consecutively for the same row.
>
> Marco
>
>   public static void increment(String tableName, String key, String family,
>                          String qualifier, int value, boolean useShort,
>                          Configuration config)
>       throws IOException{
>     HTable table = new HTable(config,
>         config.get("app", "geomon-test") + "." + tableName);
>     Get get = new Get(key.getBytes());
>     get.addColumn(family.getBytes(),
>         qualifier.getBytes());
>     int before = 0;
>     RowLock lock = table.lockRow(key.getBytes());
>     try {
>       Result result = table.get(get);
>       if (!result.isEmpty()) {
>         if (useShort) {
>           before = (int) Bytes.toShort(result.getValue(family.getBytes(),
>               qualifier.getBytes()));
>         } else {
>           before = Bytes.toInt(result.getValue(family.getBytes(),
>               qualifier.getBytes()));
>         }
>       }
>       if (useShort) {
>         value = Math.min(before + value, Short.MAX_VALUE);
>       } else {
>         value = (int) Math.min((long) before + value, Integer.MAX_VALUE);
>       }
>
>       Put put = new Put(key.getBytes());
>       put.add(family.getBytes(),
>           qualifier.getBytes(),
>           Bytes.toBytes(useShort ? (short) value : value));
>       table.put(put);
>     } catch (Exception e) {
>       System.err.println(e.getMessage());
>     } finally {
>       table.unlockRow(lock);
>     }
>   }
>
>
> 12/07/24 14:20:58 INFO mapred.JobClient: Task Id : attempt_201207241320_0002_r_000000_2, Status : FAILEDorg.apache.hadoop.hbase.UnknownRowLockException: org.apache.hadoop.hbase.UnknownRowLockException: -3110061788478328763
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)        at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>         at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
>         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
>         at org.apache.hadoop.hbase.client.ServerCallable.translateException(ServerCallable.java:228)
>         at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:166)
>         at org.apache.hadoop.hbase.client.HTable.unlockRow(HTable.java:1031)
>         at Util.increment(Util.java:170)
>         at Aggregate$EventReducer.processUserEvent(Aggregate.java:161)
>         at Aggregate$EventReducer.processUser(Aggregate.java:119)
>         at Aggregate$EventReducer.reduce(Aggregate.java:189)
>         at Aggregate$EventReducer.reduce(Aggregate.java:78)
>         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
>         at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)

I was unaware of locks not moving with regions/splits. Hmm…I just came across http://jerryjcw.blogspot.com/2009/10/hbase-notes-casual-remark-about-row.html which I'm going to try.

HBase's increment method says "readers do not take row locks so get and scan operations can see this operation partially completed", which could cause problems.

Marco  

--  
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Tuesday 24 July 2012 at 2:58 PM, Jean-Daniel Cryans wrote:

> Row locks don't move with regions or splits... are such things
> happening frequently?
>  
> Also, any reason to not use HBase's own increment method that's much
> more efficient?
>  
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#incrementColumnValue(byte[],
> byte[], byte[], long)
>  
> J-D
>  
> On Tue, Jul 24, 2012 at 2:53 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > Hi there
> >  
> > I am getting an UnknownRowLockException when adding locks to the increment() function below. The full stack trace is at the end of this message.
> >  
> > There are a few other places I am acquiring locks, but I only ever acquire a single lock for one piece of code and the rest go through fine. It's only when I enable the locks in this function that I get the exception. It's fairly simple, and I can't find anything obviously wrong with it so my best guess is there's some quirk with hbase locks that I'm unaware of. Any ideas?
> >  
> > One thing to note is that I am calling increment() a couple times consecutively for the same row.
> >  
> > Marco
> >  
> > public static void increment(String tableName, String key, String family,
> > String qualifier, int value, boolean useShort,
> > Configuration config)
> > throws IOException{
> > HTable table = new HTable(config,
> > config.get("app", "geomon-test") + "." + tableName);
> > Get get = new Get(key.getBytes());
> > get.addColumn(family.getBytes(),
> > qualifier.getBytes());
> > int before = 0;
> > RowLock lock = table.lockRow(key.getBytes());
> > try {
> > Result result = table.get(get);
> > if (!result.isEmpty()) {
> > if (useShort) {
> > before = (int) Bytes.toShort(result.getValue(family.getBytes(),
> > qualifier.getBytes()));
> > } else {
> > before = Bytes.toInt(result.getValue(family.getBytes(),
> > qualifier.getBytes()));
> > }
> > }
> > if (useShort) {
> > value = Math.min(before + value, Short.MAX_VALUE);
> > } else {
> > value = (int) Math.min((long) before + value, Integer.MAX_VALUE);
> > }
> >  
> > Put put = new Put(key.getBytes());
> > put.add(family.getBytes(),
> > qualifier.getBytes(),
> > Bytes.toBytes(useShort ? (short) value : value));
> > table.put(put);
> > } catch (Exception e) {
> > System.err.println(e.getMessage());
> > } finally {
> > table.unlockRow(lock);
> > }
> > }
> >  
> >  
> > 12/07/24 14:20:58 INFO mapred.JobClient: Task Id : attempt_201207241320_0002_r_000000_2, Status : FAILEDorg.apache.hadoop.hbase.UnknownRowLockException: org.apache.hadoop.hbase.UnknownRowLockException: -3110061788478328763
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
> > at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
> > at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
> > at org.apache.hadoop.hbase.client.ServerCallable.translateException(ServerCallable.java:228)
> > at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:166)
> > at org.apache.hadoop.hbase.client.HTable.unlockRow(HTable.java:1031)

On Tue, Jul 24, 2012 at 3:08 PM, Marco Gallotta <[EMAIL PROTECTED]> wrote:
> HBase's increment method says "readers do not take row locks so get and scan operations can see this operation partially completed", which could cause problems.

As far as I can tell your code suffers from the same issue eg if you
need to increment 2 cells you do 2 puts under two separate locks.

J-D

I acquire the lock before the get. I only setup the get before acquiring the lock.

I'm also looking into using checkAndPut.

Marco

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Tuesday 24 July 2012 at 3:13 PM, Jean-Daniel Cryans wrote:

> On Tue, Jul 24, 2012 at 3:08 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > HBase's increment method says "readers do not take row locks so get and scan operations can see this operation partially completed", which could cause problems.
>
>
> As far as I can tell your code suffers from the same issue eg if you
> need to increment 2 cells you do 2 puts under two separate locks.
>
> J-D

On Tue, Jul 24, 2012 at 3:29 PM, Marco Gallotta <[EMAIL PROTECTED]> wrote:
> I acquire the lock before the get. I only setup the get before acquiring the lock.

Yes, but let's say you want to increment "f:a" and "f:b" in the row
"example". Your code requires 2 calls to increment() and would be seen
as two different puts so a reader could see "f:a" incremented but not
"f:b". It's the same for HBase's increment function, except it's going
to be much faster and won't suffer row lock problems.

J-D

Oh, oops, I misread increments documentation. I'm fine with that. The only problem now then is that it only operates on longs, but I'm storing ints and shorts to save space. Perhaps the space saving isn't worth the cost of not using this though.

Marco

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Tuesday 24 July 2012 at 3:34 PM, Jean-Daniel Cryans wrote:

> On Tue, Jul 24, 2012 at 3:29 PM, Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > I acquire the lock before the get. I only setup the get before acquiring the lock.
>
>
> Yes, but let's say you want to increment "f:a" and "f:b" in the row
> "example". Your code requires 2 calls to increment() and would be seen
> as two different puts so a reader could see "f:a" incremented but not
> "f:b". It's the same for HBase's increment function, except it's going
> to be much faster and won't suffer row lock problems.
>
> J-D

On Tue, Jul 24, 2012 at 3:44 PM, Marco Gallotta <[EMAIL PROTECTED]> wrote:
> Oh, oops, I misread increments documentation. I'm fine with that. The only problem now then is that it only operates on longs, but I'm storing ints and shorts to save space. Perhaps the space saving isn't worth the cost of not using this though.

That's what I thought, I don't know what a partially completed
increment on a single cell would look like... see 7 while someone is
incrementing 4 by 5? :)

The space saving is a good question, are you really that short on hard
drives? The key is still the bigger part of your whole row by an order
of magnitude and this is usually where people try to shave a few bytes
off.

J-D

On Tuesday 24 July 2012 at 3:51 PM, Jean-Daniel Cryans wrote:
> The space saving is a good question, are you really that short on hard
> drives? The key is still the bigger part of your whole row by an order
> of magnitude and this is usually where people try to shave a few bytes
> off.
>
>
I have thousands of columns of per row, so I doubt that's still true?

Marco

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
 

On Tue, Jul 24, 2012 at 3:55 PM, Marco Gallotta <[EMAIL PROTECTED]> wrote:
>
> I have thousands of columns of per row, so I doubt that's still true?

No, since every value is stored along its key (row key + family +
qualifier + timestamp + length of each). That's why compression is
pretty much always recommended.

J-D