Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> hbase client RetriesExhaustedWithDetailsException with RegionTooBusyException


Copy link to this message
-
Re: hbase client RetriesExhaustedWithDetailsException with RegionTooBusyException
Busy - means busy.
Doc says:
Thrown by a region server if it will block and wait to serve a request. For
example, the client wants to insert something to a region while the
 region is compacting.

>> I have a MR job that is loading data into hbase using hbase client API.
You should avoid this. If you can't :

* Reduce # of M/R tasks.
* Disable major compaction
* Disable region splitting
* Increase max region size.
* Add retry logic to the client (M/R task).

Check bulk loading.

On Tue, Feb 4, 2014 at 5:44 PM, Jerry He <[EMAIL PROTECTED]> wrote:

> Hi, hbase experts:
>
> You probably have seen this in the past.  Can someone share some quick
> ideas?
> I have a MR job that is loading data into hbase using hbase client API.
>  The job failed after a while.  Below is the error info and stack trace.
>
> 2014-02-04 14:15:32,587 WARN org.apache.hadoop.hbase.client.AsyncProcess:
> Attempt #6/35 failed for 182 ops on hdtest203.svl.ibm.com
> ,60020,1391489734651
> NOT resubmitting.
>
> region=tpch_hb_1000.lineitem,\x01\x8Ao\x83\xF0\x01\x80'`\x04\x01\x80\x00\x00\x00\x02ufc\x01\x80\x00\x00\x01,1391544629530.b4a41acd34723629c417571b524b80ab.,
> hostname=hdtest203.svl.ibm.com,60020,1391489734651, seqNum=682710
> 2014-02-04 14:19:12,987 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=302, tasksDone=301, currentTasksDone=301,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:19:32,452 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=306, tasksDone=305, currentTasksDone=305,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:19:42,544 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=307, tasksDone=306, currentTasksDone=306,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:19:52,624 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=308, tasksDone=307, currentTasksDone=307,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:20:03,044 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=309, tasksDone=308, currentTasksDone=308,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:20:13,133 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=310, tasksDone=309, currentTasksDone=309,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:20:23,240 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=311, tasksDone=310, currentTasksDone=310,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:20:33,328 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=312, tasksDone=311, currentTasksDone=311,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:20:38,444 WARN org.apache.hadoop.hbase.client.AsyncProcess:
> Attempt #14/35 failed for 837 ops on hdtest203.svl.ibm.com
> ,60020,1391489734651
> NOT resubmitting.
>
> region=tpch_hb_1000.lineitem,\x01\x8Ao\x83\xF0\x01\x80'`\x04\x01\x80\x00\x00\x00\x02ufc\x01\x80\x00\x00\x01,1391544629530.b4a41acd34723629c417571b524b80ab.,
> hostname=hdtest203.svl.ibm.com,60020,1391489734651, seqNum=682710
> 2014-02-04 14:20:38,452 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2014-02-04 14:20:38,472 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:hive (auth:SIMPLE)
> cause:org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed 837 actions: RegionTooBusyException: 837 times,

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB