Just checked and there are a lot of 'not balancing because there are unhosted tablets' debug messages. Is this the same issue?
From: Josh Elser [mailto:[EMAIL PROTECTED]]
Sent: Monday, 2 September 2013 12:52
To: [EMAIL PROTECTED]
Subject: Re: High Ingest on a single server [SEC=UNOFFICIAL]
To verify that this what you're running into, you should see a message in the master*.debug.log that matches "not balancing because.*"
On 09/01/2013 09:54 PM, John Vines wrote:
> Try restarting the master. A few releases had a big where it would get
> Sent from my phone, please pardon the typos and brevity.
> On Sep 1, 2013 6:12 PM, "Dickson, Matt MR"
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
> Thanks Eric.
> The tablet of concern has 2000 tablets while all others have 1000 so
> it looks like the balancers aren't evening out the tablets per node
> as expected.
> Is there a way to force the balancer to run or rectify this
> situation by moving tablets to alternate nodes?
> *From:* Eric Newton [mailto:[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>]
> *Sent:* Thursday, 29 August 2013 23:23
> *To:* [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> *Subject:* Re: High Ingest on a single server [SEC=UNOFFICIAL]
> The balancers that ship with accumulo attempt to keep an equal
> number of tablets on each server. An empty tablet, will be balanced
> with the same weight as a 50G tablet.
> You can write a new balancer to take advantage of the properties of
> the tablets, and any expected hotspots you have.
> On Thu, Aug 29, 2013 at 1:39 AM, Dickson, Matt MR
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> We are seeing a single server that has less entries than all the
> other nodes in the cluster. Accumulo now appears to be
> directing higher ingest tablets to this node and its now getting
> 7 times the ingest entries than all other nodes and is slowing
> or load. Does Accumulo attempt to balance disk usage across the
> nodes for a table by moving tablets and that is why we are
> seeing this node ingesting more?
> If not, is it possible to make accumulo rebalance the ingest
> across all servers during a load?