Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo >> mail # dev >> ingest performance oscillations and Xceivers


+
Kepner, Jeremy - 1010 - M... 2013-01-02, 02:19
Copy link to this message
-
Re: ingest performance oscillations and Xceivers
Hey Jeremy,

Can you compare the ingest rate to the number of tablets, too?

I've found, that if I have 20-80 tablets per server (on similar hardware) I
get the best performance.

# of Xceivers == number of writers when ingest is the primary target.

Also, is this 1.4 or trunk?

-Eric

On Tue, Jan 1, 2013 at 9:19 PM, Kepner, Jeremy - 1010 - MITLL <
[EMAIL PROTECTED]> wrote:

> Accumulo Colleagues,
>   I am trying to optimize my ingest into a single node Accumulo instance
> running on a 32 core node with 96 GB of RAM.  I am seeing the follow ingest
> variations as a I change the number of ingest processes (see attached):
>
> -------------------------------------
> Ingestors, Ingest rate
> -------------------------------------
> 1, 60K inserts/sec (stable)
> 2, 120K inserts/sec (stable)
> 3, 60K to 180K inserts/sec
> 4, 90K to 220K inserts/sec
> 8, 80K to 280K inserts/sec
> 12, 80K to 280K inserts/sec
> -------------------------------------
>
> The only thing I can see that correlates with the ingest rate is the
> number of Xceivers.  When the ingest rate is high the number of Xceivers is
> usually low.  Likewise, when the ingest rate drops, the number of Xceivers
> usually increases significantly.
>
> Question: What role to Xceivers play in ingest?
>
> Request: It would be great to add a plot showing the number of Xceivers
> over time to the diagnostics.
>
> Regards.  -Jeremy
>
>
+
Jeremy Kepner 2013-01-02, 19:11
+
William Slacum 2013-01-02, 12:09
+
Kepner, Jeremy - 0553 - M... 2013-01-03, 00:25
+
Eric Newton 2013-01-03, 18:19
+
William Slacum 2013-01-03, 16:24
+
Jeremy Kepner 2013-01-03, 22:25
+
John Vines 2013-01-03, 23:04
+
Kepner, Jeremy - 0553 - M... 2013-01-04, 00:31
+
Josh Elser 2013-01-04, 00:47
+
Kepner, Jeremy - 0553 - M... 2013-01-04, 00:55
+
Drew Farris 2013-01-04, 01:26
+
Kepner, Jeremy - 0553 - M... 2013-01-04, 16:30
+
John Vines 2013-01-04, 01:10
+
Josh Elser 2013-01-04, 01:10
+
Josh Elser 2013-01-04, 01:13