Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Hbase thrift server crashing on writes


+
Varun Sharma 2012-11-06, 23:12
Copy link to this message
-
Re: Hbase thrift server crashing on writes
Just One more thing - this is still happening and its happening only for a
certain set of region servers and regions only. Also, its triggered by
mutateRow operations. Is this a data corruption issue - we are running on
top of HDFS with replication factor of 2.

Thanks
VArun

On Tue, Nov 6, 2012 at 3:12 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am seeing thrift crashes which seem to be originating due to some issues
> at the region servers. Here is the log - this looks quite concerning since
> the server is failing to acquire a lock even when blocking - what is going
> on here ?
>
> 12/11/06 21:53:29 WARN
> client.HConnectionManager$HConnectionImplementation: Failed all from
> region=user_likes_pins_ts_tall,0025c0300000074b7fffffffb0563d20010f4010000d6d6a,1351650405902.7d695b6f2b9e13e97c8bde70641c8aee.,
> hostname=ip-10-31-190-107.ec2.internal, port=60020
> java.util.concurrent.ExecutionException: java.io.IOException:
> java.io.IOException: java.lang.AssertionError: Should never fail to get
> lock when blocking
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1850)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1755)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3109)
>         at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:369)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
>
>         at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>         at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1558)
>         at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1410)
>         at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:943)
>         at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:820)
>         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:795)
>         at
> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.mutateRowTs(ThriftServer.java:613)
>         at
> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.mutateRow(ThriftServer.java:576)
>         at
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$mutateRow.getResult(Hbase.java:3630)
>         at
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$mutateRow.getResult(Hbase.java:3618)
>         at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
>         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
>         at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.AssertionError: Should never fail to get lock when blocking
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1850)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1755)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3109)
>         at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:369)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB