Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # dev >> ingest performance oscillations and Xceivers


Copy link to this message
-
Re: ingest performance oscillations and Xceivers
Have you also been tracking compactions? Did you have a query load?
On Wed, Jan 2, 2013 at 7:25 PM, Kepner, Jeremy - 0553 - MITLL <
[EMAIL PROTECTED]> wrote:

> Hmmm, that's interesting, because in the past I didn't see this behavior.
>  It might be worth having someone look into because it seems to have a 2x
> impact on sustained ingest.
>
> Regards.  -Jeremy
>
> On Jan 2, 2013, at 2:23 PM, Keith Turner wrote:
>
> > On Wed, Jan 2, 2013 at 2:11 PM, Jeremy Kepner <[EMAIL PROTECTED]> wrote:
> >> So what mechanism causes the number of Xceivers to increase?
> >
> > Its been a while since I looked at the data node source code.   When I
> > last look at it an Xceiver was just a thread created to handle a
> > datanode request.   The thread went away after the request was
> > processed.   So major and minor compactions running would cause more
> > Xceivers to be created to read and write data.
> >
> > Newer datanode code may use a thread pool instead of creating a
> > thread/xceiver for each request.   I am not sure.
> >
> >> I am carefully controlling the number of ingestors and the data isn't
> varying too much.
> >> I would expect the number of Xceivers to remain consant.
> >>
> >> Regards.  -Jeremy
> >>
> >> On Tue, Jan 01, 2013 at 09:45:20PM -0500, Eric Newton wrote:
> >>> Hey Jeremy,
> >>>
> >>> Can you compare the ingest rate to the number of tablets, too?
> >>>
> >>> I've found, that if I have 20-80 tablets per server (on similar
> hardware) I
> >>> get the best performance.
> >>>
> >>> # of Xceivers == number of writers when ingest is the primary target.
> >>>
> >>> Also, is this 1.4 or trunk?
> >>>
> >>> -Eric
> >>>
> >>>
> >>>
> >>> On Tue, Jan 1, 2013 at 9:19 PM, Kepner, Jeremy - 1010 - MITLL <
> >>> [EMAIL PROTECTED]> wrote:
> >>>
> >>>> Accumulo Colleagues,
> >>>>  I am trying to optimize my ingest into a single node Accumulo
> instance
> >>>> running on a 32 core node with 96 GB of RAM.  I am seeing the follow
> ingest
> >>>> variations as a I change the number of ingest processes (see
> attached):
> >>>>
> >>>> -------------------------------------
> >>>> Ingestors, Ingest rate
> >>>> -------------------------------------
> >>>> 1, 60K inserts/sec (stable)
> >>>> 2, 120K inserts/sec (stable)
> >>>> 3, 60K to 180K inserts/sec
> >>>> 4, 90K to 220K inserts/sec
> >>>> 8, 80K to 280K inserts/sec
> >>>> 12, 80K to 280K inserts/sec
> >>>> -------------------------------------
> >>>>
> >>>> The only thing I can see that correlates with the ingest rate is the
> >>>> number of Xceivers.  When the ingest rate is high the number of
> Xceivers is
> >>>> usually low.  Likewise, when the ingest rate drops, the number of
> Xceivers
> >>>> usually increases significantly.
> >>>>
> >>>> Question: What role to Xceivers play in ingest?
> >>>>
> >>>> Request: It would be great to add a plot showing the number of
> Xceivers
> >>>> over time to the diagnostics.
> >>>>
> >>>> Regards.  -Jeremy
> >>>>
> >>>>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB