Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo, mail # dev - ingest performance oscillations and Xceivers


+
Kepner, Jeremy - 1010 - M... 2013-01-02, 02:19
+
Eric Newton 2013-01-02, 02:45
+
Jeremy Kepner 2013-01-02, 19:11
+
William Slacum 2013-01-02, 12:09
+
Kepner, Jeremy - 0553 - M... 2013-01-03, 00:25
+
Eric Newton 2013-01-03, 18:19
Copy link to this message
-
Re: ingest performance oscillations and Xceivers
William Slacum 2013-01-03, 16:24
Have you also been tracking compactions? Did you have a query load?
On Wed, Jan 2, 2013 at 7:25 PM, Kepner, Jeremy - 0553 - MITLL <
[EMAIL PROTECTED]> wrote:

> Hmmm, that's interesting, because in the past I didn't see this behavior.
>  It might be worth having someone look into because it seems to have a 2x
> impact on sustained ingest.
>
> Regards.  -Jeremy
>
> On Jan 2, 2013, at 2:23 PM, Keith Turner wrote:
>
> > On Wed, Jan 2, 2013 at 2:11 PM, Jeremy Kepner <[EMAIL PROTECTED]> wrote:
> >> So what mechanism causes the number of Xceivers to increase?
> >
> > Its been a while since I looked at the data node source code.   When I
> > last look at it an Xceiver was just a thread created to handle a
> > datanode request.   The thread went away after the request was
> > processed.   So major and minor compactions running would cause more
> > Xceivers to be created to read and write data.
> >
> > Newer datanode code may use a thread pool instead of creating a
> > thread/xceiver for each request.   I am not sure.
> >
> >> I am carefully controlling the number of ingestors and the data isn't
> varying too much.
> >> I would expect the number of Xceivers to remain consant.
> >>
> >> Regards.  -Jeremy
> >>
> >> On Tue, Jan 01, 2013 at 09:45:20PM -0500, Eric Newton wrote:
> >>> Hey Jeremy,
> >>>
> >>> Can you compare the ingest rate to the number of tablets, too?
> >>>
> >>> I've found, that if I have 20-80 tablets per server (on similar
> hardware) I
> >>> get the best performance.
> >>>
> >>> # of Xceivers == number of writers when ingest is the primary target.
> >>>
> >>> Also, is this 1.4 or trunk?
> >>>
> >>> -Eric
> >>>
> >>>
> >>>
> >>> On Tue, Jan 1, 2013 at 9:19 PM, Kepner, Jeremy - 1010 - MITLL <
> >>> [EMAIL PROTECTED]> wrote:
> >>>
> >>>> Accumulo Colleagues,
> >>>>  I am trying to optimize my ingest into a single node Accumulo
> instance
> >>>> running on a 32 core node with 96 GB of RAM.  I am seeing the follow
> ingest
> >>>> variations as a I change the number of ingest processes (see
> attached):
> >>>>
> >>>> -------------------------------------
> >>>> Ingestors, Ingest rate
> >>>> -------------------------------------
> >>>> 1, 60K inserts/sec (stable)
> >>>> 2, 120K inserts/sec (stable)
> >>>> 3, 60K to 180K inserts/sec
> >>>> 4, 90K to 220K inserts/sec
> >>>> 8, 80K to 280K inserts/sec
> >>>> 12, 80K to 280K inserts/sec
> >>>> -------------------------------------
> >>>>
> >>>> The only thing I can see that correlates with the ingest rate is the
> >>>> number of Xceivers.  When the ingest rate is high the number of
> Xceivers is
> >>>> usually low.  Likewise, when the ingest rate drops, the number of
> Xceivers
> >>>> usually increases significantly.
> >>>>
> >>>> Question: What role to Xceivers play in ingest?
> >>>>
> >>>> Request: It would be great to add a plot showing the number of
> Xceivers
> >>>> over time to the diagnostics.
> >>>>
> >>>> Regards.  -Jeremy
> >>>>
> >>>>
>
>
+
Jeremy Kepner 2013-01-03, 22:25
+
John Vines 2013-01-03, 23:04
+
Kepner, Jeremy - 0553 - M... 2013-01-04, 00:31
+
Josh Elser 2013-01-04, 00:47
+
Kepner, Jeremy - 0553 - M... 2013-01-04, 00:55
+
Drew Farris 2013-01-04, 01:26
+
Kepner, Jeremy - 0553 - M... 2013-01-04, 16:30
+
John Vines 2013-01-04, 01:10
+
Josh Elser 2013-01-04, 01:10
+
Josh Elser 2013-01-04, 01:13