Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> hbase client RetriesExhaustedWithDetailsException with RegionTooBusyException


Copy link to this message
-
Re: hbase client RetriesExhaustedWithDetailsException with RegionTooBusyException
Busy - means busy.
Doc says:
Thrown by a region server if it will block and wait to serve a request. For
example, the client wants to insert something to a region while the
 region is compacting.

>> I have a MR job that is loading data into hbase using hbase client API.
You should avoid this. If you can't :

* Reduce # of M/R tasks.
* Disable major compaction
* Disable region splitting
* Increase max region size.
* Add retry logic to the client (M/R task).

Check bulk loading.

On Tue, Feb 4, 2014 at 5:44 PM, Jerry He <[EMAIL PROTECTED]> wrote:

> Hi, hbase experts:
>
> You probably have seen this in the past.  Can someone share some quick
> ideas?
> I have a MR job that is loading data into hbase using hbase client API.
>  The job failed after a while.  Below is the error info and stack trace.
>
> 2014-02-04 14:15:32,587 WARN org.apache.hadoop.hbase.client.AsyncProcess:
> Attempt #6/35 failed for 182 ops on hdtest203.svl.ibm.com
> ,60020,1391489734651
> NOT resubmitting.
>
> region=tpch_hb_1000.lineitem,\x01\x8Ao\x83\xF0\x01\x80'`\x04\x01\x80\x00\x00\x00\x02ufc\x01\x80\x00\x00\x01,1391544629530.b4a41acd34723629c417571b524b80ab.,
> hostname=hdtest203.svl.ibm.com,60020,1391489734651, seqNum=682710
> 2014-02-04 14:19:12,987 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=302, tasksDone=301, currentTasksDone=301,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:19:32,452 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=306, tasksDone=305, currentTasksDone=305,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:19:42,544 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=307, tasksDone=306, currentTasksDone=306,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:19:52,624 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=308, tasksDone=307, currentTasksDone=307,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:20:03,044 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=309, tasksDone=308, currentTasksDone=308,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:20:13,133 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=310, tasksDone=309, currentTasksDone=309,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:20:23,240 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=311, tasksDone=310, currentTasksDone=310,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:20:33,328 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
> Waiting for the global number of running tasks to be equals or less than 0,
> tasksSent=312, tasksDone=311, currentTasksDone=311,
> tableName=tpch_hb_1000.lineitem
> 2014-02-04 14:20:38,444 WARN org.apache.hadoop.hbase.client.AsyncProcess:
> Attempt #14/35 failed for 837 ops on hdtest203.svl.ibm.com
> ,60020,1391489734651
> NOT resubmitting.
>
> region=tpch_hb_1000.lineitem,\x01\x8Ao\x83\xF0\x01\x80'`\x04\x01\x80\x00\x00\x00\x02ufc\x01\x80\x00\x00\x01,1391544629530.b4a41acd34723629c417571b524b80ab.,
> hostname=hdtest203.svl.ibm.com,60020,1391489734651, seqNum=682710
> 2014-02-04 14:20:38,452 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2014-02-04 14:20:38,472 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:hive (auth:SIMPLE)
> cause:org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed 837 actions: RegionTooBusyException: 837 times,