Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo >> mail # dev >> ingest performance oscillations and Xceivers


+
Kepner, Jeremy - 1010 - M... 2013-01-02, 02:19
+
Eric Newton 2013-01-02, 02:45
+
Jeremy Kepner 2013-01-02, 19:11
+
William Slacum 2013-01-02, 12:09
+
Kepner, Jeremy - 0553 - M... 2013-01-03, 00:25
+
Eric Newton 2013-01-03, 18:19
+
William Slacum 2013-01-03, 16:24
Copy link to this message
-
Re: ingest performance oscillations and Xceivers
No correlation with compactions.  No queries.

On Thu, Jan 03, 2013 at 11:24:17AM -0500, William Slacum wrote:
> Have you also been tracking compactions? Did you have a query load?
>
>
> On Wed, Jan 2, 2013 at 7:25 PM, Kepner, Jeremy - 0553 - MITLL <
> [EMAIL PROTECTED]> wrote:
>
> > Hmmm, that's interesting, because in the past I didn't see this behavior.
> >  It might be worth having someone look into because it seems to have a 2x
> > impact on sustained ingest.
> >
> > Regards.  -Jeremy
> >
> > On Jan 2, 2013, at 2:23 PM, Keith Turner wrote:
> >
> > > On Wed, Jan 2, 2013 at 2:11 PM, Jeremy Kepner <[EMAIL PROTECTED]> wrote:
> > >> So what mechanism causes the number of Xceivers to increase?
> > >
> > > Its been a while since I looked at the data node source code.   When I
> > > last look at it an Xceiver was just a thread created to handle a
> > > datanode request.   The thread went away after the request was
> > > processed.   So major and minor compactions running would cause more
> > > Xceivers to be created to read and write data.
> > >
> > > Newer datanode code may use a thread pool instead of creating a
> > > thread/xceiver for each request.   I am not sure.
> > >
> > >> I am carefully controlling the number of ingestors and the data isn't
> > varying too much.
> > >> I would expect the number of Xceivers to remain consant.
> > >>
> > >> Regards.  -Jeremy
> > >>
> > >> On Tue, Jan 01, 2013 at 09:45:20PM -0500, Eric Newton wrote:
> > >>> Hey Jeremy,
> > >>>
> > >>> Can you compare the ingest rate to the number of tablets, too?
> > >>>
> > >>> I've found, that if I have 20-80 tablets per server (on similar
> > hardware) I
> > >>> get the best performance.
> > >>>
> > >>> # of Xceivers == number of writers when ingest is the primary target.
> > >>>
> > >>> Also, is this 1.4 or trunk?
> > >>>
> > >>> -Eric
> > >>>
> > >>>
> > >>>
> > >>> On Tue, Jan 1, 2013 at 9:19 PM, Kepner, Jeremy - 1010 - MITLL <
> > >>> [EMAIL PROTECTED]> wrote:
> > >>>
> > >>>> Accumulo Colleagues,
> > >>>>  I am trying to optimize my ingest into a single node Accumulo
> > instance
> > >>>> running on a 32 core node with 96 GB of RAM.  I am seeing the follow
> > ingest
> > >>>> variations as a I change the number of ingest processes (see
> > attached):
> > >>>>
> > >>>> -------------------------------------
> > >>>> Ingestors, Ingest rate
> > >>>> -------------------------------------
> > >>>> 1, 60K inserts/sec (stable)
> > >>>> 2, 120K inserts/sec (stable)
> > >>>> 3, 60K to 180K inserts/sec
> > >>>> 4, 90K to 220K inserts/sec
> > >>>> 8, 80K to 280K inserts/sec
> > >>>> 12, 80K to 280K inserts/sec
> > >>>> -------------------------------------
> > >>>>
> > >>>> The only thing I can see that correlates with the ingest rate is the
> > >>>> number of Xceivers.  When the ingest rate is high the number of
> > Xceivers is
> > >>>> usually low.  Likewise, when the ingest rate drops, the number of
> > Xceivers
> > >>>> usually increases significantly.
> > >>>>
> > >>>> Question: What role to Xceivers play in ingest?
> > >>>>
> > >>>> Request: It would be great to add a plot showing the number of
> > Xceivers
> > >>>> over time to the diagnostics.
> > >>>>
> > >>>> Regards.  -Jeremy
> > >>>>
> > >>>>
> >
> >
+
John Vines 2013-01-03, 23:04
+
Kepner, Jeremy - 0553 - M... 2013-01-04, 00:31
+
Josh Elser 2013-01-04, 00:47
+
Kepner, Jeremy - 0553 - M... 2013-01-04, 00:55
+
Drew Farris 2013-01-04, 01:26
+
Kepner, Jeremy - 0553 - M... 2013-01-04, 16:30
+
John Vines 2013-01-04, 01:10
+
Josh Elser 2013-01-04, 01:10
+
Josh Elser 2013-01-04, 01:13