Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # dev >> GSOC: Monitor Improvements


Copy link to this message
-
Re: GSOC: Monitor Improvements
I would do something simpler: just have a Mock collector which does no JMX,
it just makes up numbers, which could be substituted for testing.

-Eric

On Mon, Apr 22, 2013 at 11:04 AM, Supun Kamburugamuva <[EMAIL PROTECTED]>wrote:

> That sounds interesting. To clarify the requirement, we can have a process
> that exposes the same JMX mbeans as the the real server and monitor can
> plug in to this process.
>
> Thanks,
> Supun..
>
>
> On Mon, Apr 22, 2013 at 10:57 AM, Josh Elser <[EMAIL PROTECTED]> wrote:
>
> > That would be pretty sweet, actually. Potentially parallel to what you
> > want to do, Supun, but cool nonetheless.
> >
> > I could see a lot of benefit by having some process that could emulate
> the
> > output from a non-trivially-sized Accumulo cluster on a single box.
> >
> >
> > On 4/22/13 10:43 AM, Eric Newton wrote:
> >
> >> You could mock the stats collection.
> >>
> >> -Eric
> >>
> >>
> >> On Mon, Apr 22, 2013 at 10:41 AM, David Medinets
> >> <[EMAIL PROTECTED]>**wrote:
> >>
> >>  The average developer probably can't access a large cluster with
> hundred
> >>> of
> >>> nodes. Is there a way to simulate this?
> >>>
> >>>
> >>> On Mon, Apr 22, 2013 at 9:05 AM, Eric Newton <[EMAIL PROTECTED]>
> >>> wrote:
> >>>
> >>>  Another thing to consider is scale.  On large clusters (many hundreds
> of
> >>>> nodes), more data is not helpful for visualization.  Instead,
> summaries,
> >>>> averages and outliers are important.
> >>>>
> >>>> For example, if one node is consistently slow, it is better to know
> that
> >>>> than to see one graph with low numbers in a sea of graphs.
> >>>>
> >>>> If the monitor collects information using JMX, collection time for
> each
> >>>> node would be a good thing to know, too.
> >>>>
> >>>> -Eric
> >>>>
> >>>>
> >>>> On Sun, Apr 21, 2013 at 10:00 PM, Josh Elser <[EMAIL PROTECTED]>
> >>>>
> >>> wrote:
> >>>
> >>>> Supun,
> >>>>>
> >>>>> Yup, very much so. Having a way to consume any and all metrics via
> JMX
> >>>>> would simplify things for any consumers (internal or external).
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 04/21/2013 02:15 PM, Supun Kamburugamuva wrote:
> >>>>>
> >>>>>  Hi Josh,
> >>>>>>
> >>>>>> Thanks for the suggestions. I'll incorporate these to the proposal.
> >>>>>>
> >>>>>> Another area I would like to work is on JMX. There is a Jira that
> says
> >>>>>>
> >>>>> to
> >>>>
> >>>>> replace the Monitor calls from Thrift to JMX (Accumulo 694). Do you
> >>>>>>
> >>>>> think
> >>>>
> >>>>> this is a good addition to the Monitor?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Supun..
> >>>>>>
> >>>>>>
> >>>>>> On Sun, Apr 21, 2013 at 1:45 PM, Josh Elser <[EMAIL PROTECTED]>
> >>>>>>
> >>>>> wrote:
> >>>>
> >>>>>   Supun,
> >>>>>>
> >>>>>>> Looks good! Can I make some suggestions/comments?
> >>>>>>>
> >>>>>>> For: "Per table plots: ACCUMULO-594", I'd also like to see minor
> >>>>>>> compactions, major compactions, index cache hit rate, and data
> cache
> >>>>>>>
> >>>>>> hit
> >>>>
> >>>>> rate per table (same graphs that are displayed system-wide when you
> >>>>>>>
> >>>>>> visit
> >>>>
> >>>>> http://${MONITOR_HOST}:50095/.
> >>>>>>>
> >>>>>>> For "Per tablet [server] plots", it would be neat if you could also
> >>>>>>> extract some general statistics like top N least performing, top N
> >>>>>>> highest
> >>>>>>> performing, etc. tablet servers. Ideally, this could correlate with
> >>>>>>> servers
> >>>>>>> that may be having problems :).
> >>>>>>>
> >>>>>>> Do you see these proposed changes as being sufficient for 3-4
> months
> >>>>>>>
> >>>>>> of
> >>>
> >>>>  40hrs/week work? If you plan to really dig into these changes
> >>>>>>>
> >>>>>> (perhaps
> >>>
> >>>>  reworking components of the monitor itself), I could perhaps see
> >>>>>>>
> >>>>>> this.
> >>>
> >>>> Do
> >>>>
> >>>>> you have any ideas for more lofty goals that you could pursue as
> >>>>>>>
> >>>>>> well?
> >>>
> >>>> I
> >>>>
> >>>>> don't want you/us to get one month into things and see you complete
> >>>>>>