Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # dev >> GSOC: Monitor Improvements


Copy link to this message
-
Re: GSOC: Monitor Improvements
I would do something simpler: just have a Mock collector which does no JMX,
it just makes up numbers, which could be substituted for testing.

-Eric

On Mon, Apr 22, 2013 at 11:04 AM, Supun Kamburugamuva <[EMAIL PROTECTED]>wrote:

> That sounds interesting. To clarify the requirement, we can have a process
> that exposes the same JMX mbeans as the the real server and monitor can
> plug in to this process.
>
> Thanks,
> Supun..
>
>
> On Mon, Apr 22, 2013 at 10:57 AM, Josh Elser <[EMAIL PROTECTED]> wrote:
>
> > That would be pretty sweet, actually. Potentially parallel to what you
> > want to do, Supun, but cool nonetheless.
> >
> > I could see a lot of benefit by having some process that could emulate
> the
> > output from a non-trivially-sized Accumulo cluster on a single box.
> >
> >
> > On 4/22/13 10:43 AM, Eric Newton wrote:
> >
> >> You could mock the stats collection.
> >>
> >> -Eric
> >>
> >>
> >> On Mon, Apr 22, 2013 at 10:41 AM, David Medinets
> >> <[EMAIL PROTECTED]>**wrote:
> >>
> >>  The average developer probably can't access a large cluster with
> hundred
> >>> of
> >>> nodes. Is there a way to simulate this?
> >>>
> >>>
> >>> On Mon, Apr 22, 2013 at 9:05 AM, Eric Newton <[EMAIL PROTECTED]>
> >>> wrote:
> >>>
> >>>  Another thing to consider is scale.  On large clusters (many hundreds
> of
> >>>> nodes), more data is not helpful for visualization.  Instead,
> summaries,
> >>>> averages and outliers are important.
> >>>>
> >>>> For example, if one node is consistently slow, it is better to know
> that
> >>>> than to see one graph with low numbers in a sea of graphs.
> >>>>
> >>>> If the monitor collects information using JMX, collection time for
> each
> >>>> node would be a good thing to know, too.
> >>>>
> >>>> -Eric
> >>>>
> >>>>
> >>>> On Sun, Apr 21, 2013 at 10:00 PM, Josh Elser <[EMAIL PROTECTED]>
> >>>>
> >>> wrote:
> >>>
> >>>> Supun,
> >>>>>
> >>>>> Yup, very much so. Having a way to consume any and all metrics via
> JMX
> >>>>> would simplify things for any consumers (internal or external).
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 04/21/2013 02:15 PM, Supun Kamburugamuva wrote:
> >>>>>
> >>>>>  Hi Josh,
> >>>>>>
> >>>>>> Thanks for the suggestions. I'll incorporate these to the proposal.
> >>>>>>
> >>>>>> Another area I would like to work is on JMX. There is a Jira that
> says
> >>>>>>
> >>>>> to
> >>>>
> >>>>> replace the Monitor calls from Thrift to JMX (Accumulo 694). Do you
> >>>>>>
> >>>>> think
> >>>>
> >>>>> this is a good addition to the Monitor?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Supun..
> >>>>>>
> >>>>>>
> >>>>>> On Sun, Apr 21, 2013 at 1:45 PM, Josh Elser <[EMAIL PROTECTED]>
> >>>>>>
> >>>>> wrote:
> >>>>
> >>>>>   Supun,
> >>>>>>
> >>>>>>> Looks good! Can I make some suggestions/comments?
> >>>>>>>
> >>>>>>> For: "Per table plots: ACCUMULO-594", I'd also like to see minor
> >>>>>>> compactions, major compactions, index cache hit rate, and data
> cache
> >>>>>>>
> >>>>>> hit
> >>>>
> >>>>> rate per table (same graphs that are displayed system-wide when you
> >>>>>>>
> >>>>>> visit
> >>>>
> >>>>> http://${MONITOR_HOST}:50095/.
> >>>>>>>
> >>>>>>> For "Per tablet [server] plots", it would be neat if you could also
> >>>>>>> extract some general statistics like top N least performing, top N
> >>>>>>> highest
> >>>>>>> performing, etc. tablet servers. Ideally, this could correlate with
> >>>>>>> servers
> >>>>>>> that may be having problems :).
> >>>>>>>
> >>>>>>> Do you see these proposed changes as being sufficient for 3-4
> months
> >>>>>>>
> >>>>>> of
> >>>
> >>>>  40hrs/week work? If you plan to really dig into these changes
> >>>>>>>
> >>>>>> (perhaps
> >>>
> >>>>  reworking components of the monitor itself), I could perhaps see
> >>>>>>>
> >>>>>> this.
> >>>
> >>>> Do
> >>>>
> >>>>> you have any ideas for more lofty goals that you could pursue as
> >>>>>>>
> >>>>>> well?
> >>>
> >>>> I
> >>>>
> >>>>> don't want you/us to get one month into things and see you complete
> >>>>>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB