Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo >> mail # dev >> GSOC: Monitor Improvements


+
Supun Kamburugamuva 2013-04-21, 14:37
+
Josh Elser 2013-04-21, 17:45
+
Supun Kamburugamuva 2013-04-21, 18:15
+
Josh Elser 2013-04-22, 02:00
+
Eric Newton 2013-04-22, 13:05
+
Mike Drob 2013-04-22, 15:42
+
Keith Turner 2013-04-22, 16:02
+
Supun Kamburugamuva 2013-04-22, 16:42
+
Keith Turner 2013-04-22, 17:50
+
Miguel Pereira 2013-07-24, 14:02
+
David Medinets 2013-04-22, 14:41
+
Eric Newton 2013-04-22, 14:43
+
Josh Elser 2013-04-22, 14:57
+
Supun Kamburugamuva 2013-04-22, 15:04
+
Eric Newton 2013-04-22, 15:17
Copy link to this message
-
Re: GSOC: Monitor Improvements
Thank you all for the valuable input. I'll start writing the proposal. I
really like to contribute to Accumulo and would like to take on the RRDTool
proposal by Eric after the summer. Hopefully I'll have time.

Thanks,
Supun..
On Mon, Apr 22, 2013 at 11:17 AM, Eric Newton <[EMAIL PROTECTED]> wrote:

> I would do something simpler: just have a Mock collector which does no JMX,
> it just makes up numbers, which could be substituted for testing.
>
> -Eric
>
>
>
> On Mon, Apr 22, 2013 at 11:04 AM, Supun Kamburugamuva <[EMAIL PROTECTED]
> >wrote:
>
> > That sounds interesting. To clarify the requirement, we can have a
> process
> > that exposes the same JMX mbeans as the the real server and monitor can
> > plug in to this process.
> >
> > Thanks,
> > Supun..
> >
> >
> > On Mon, Apr 22, 2013 at 10:57 AM, Josh Elser <[EMAIL PROTECTED]>
> wrote:
> >
> > > That would be pretty sweet, actually. Potentially parallel to what you
> > > want to do, Supun, but cool nonetheless.
> > >
> > > I could see a lot of benefit by having some process that could emulate
> > the
> > > output from a non-trivially-sized Accumulo cluster on a single box.
> > >
> > >
> > > On 4/22/13 10:43 AM, Eric Newton wrote:
> > >
> > >> You could mock the stats collection.
> > >>
> > >> -Eric
> > >>
> > >>
> > >> On Mon, Apr 22, 2013 at 10:41 AM, David Medinets
> > >> <[EMAIL PROTECTED]>**wrote:
> > >>
> > >>  The average developer probably can't access a large cluster with
> > hundred
> > >>> of
> > >>> nodes. Is there a way to simulate this?
> > >>>
> > >>>
> > >>> On Mon, Apr 22, 2013 at 9:05 AM, Eric Newton <[EMAIL PROTECTED]>
> > >>> wrote:
> > >>>
> > >>>  Another thing to consider is scale.  On large clusters (many
> hundreds
> > of
> > >>>> nodes), more data is not helpful for visualization.  Instead,
> > summaries,
> > >>>> averages and outliers are important.
> > >>>>
> > >>>> For example, if one node is consistently slow, it is better to know
> > that
> > >>>> than to see one graph with low numbers in a sea of graphs.
> > >>>>
> > >>>> If the monitor collects information using JMX, collection time for
> > each
> > >>>> node would be a good thing to know, too.
> > >>>>
> > >>>> -Eric
> > >>>>
> > >>>>
> > >>>> On Sun, Apr 21, 2013 at 10:00 PM, Josh Elser <[EMAIL PROTECTED]>
> > >>>>
> > >>> wrote:
> > >>>
> > >>>> Supun,
> > >>>>>
> > >>>>> Yup, very much so. Having a way to consume any and all metrics via
> > JMX
> > >>>>> would simplify things for any consumers (internal or external).
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On 04/21/2013 02:15 PM, Supun Kamburugamuva wrote:
> > >>>>>
> > >>>>>  Hi Josh,
> > >>>>>>
> > >>>>>> Thanks for the suggestions. I'll incorporate these to the
> proposal.
> > >>>>>>
> > >>>>>> Another area I would like to work is on JMX. There is a Jira that
> > says
> > >>>>>>
> > >>>>> to
> > >>>>
> > >>>>> replace the Monitor calls from Thrift to JMX (Accumulo 694). Do you
> > >>>>>>
> > >>>>> think
> > >>>>
> > >>>>> this is a good addition to the Monitor?
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Supun..
> > >>>>>>
> > >>>>>>
> > >>>>>> On Sun, Apr 21, 2013 at 1:45 PM, Josh Elser <[EMAIL PROTECTED]
> >
> > >>>>>>
> > >>>>> wrote:
> > >>>>
> > >>>>>   Supun,
> > >>>>>>
> > >>>>>>> Looks good! Can I make some suggestions/comments?
> > >>>>>>>
> > >>>>>>> For: "Per table plots: ACCUMULO-594", I'd also like to see minor
> > >>>>>>> compactions, major compactions, index cache hit rate, and data
> > cache
> > >>>>>>>
> > >>>>>> hit
> > >>>>
> > >>>>> rate per table (same graphs that are displayed system-wide when you
> > >>>>>>>
> > >>>>>> visit
> > >>>>
> > >>>>> http://${MONITOR_HOST}:50095/.
> > >>>>>>>
> > >>>>>>> For "Per tablet [server] plots", it would be neat if you could
> also
> > >>>>>>> extract some general statistics like top N least performing, top
> N
> > >>>>>>> highest
> > >>>>>>> performing, etc. tablet servers. Ideally, this could correlate
> with
> > >>>>>>> servers
> > >>>>>>> that may be having problems :).

Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: [EMAIL PROTECTED];  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
+
Supun Kamburugamuva 2013-04-22, 14:33
+
Eric Newton 2013-04-22, 15:03
+
Gabe Bell 2013-04-22, 15:09