Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo, mail # dev - GSOC: Monitor Improvements


+
Supun Kamburugamuva 2013-04-21, 14:37
+
Josh Elser 2013-04-21, 17:45
+
Supun Kamburugamuva 2013-04-21, 18:15
+
Josh Elser 2013-04-22, 02:00
+
Eric Newton 2013-04-22, 13:05
+
Mike Drob 2013-04-22, 15:42
+
Keith Turner 2013-04-22, 16:02
+
Supun Kamburugamuva 2013-04-22, 16:42
+
Keith Turner 2013-04-22, 17:50
+
Miguel Pereira 2013-07-24, 14:02
+
David Medinets 2013-04-22, 14:41
+
Eric Newton 2013-04-22, 14:43
+
Josh Elser 2013-04-22, 14:57
+
Supun Kamburugamuva 2013-04-22, 15:04
+
Eric Newton 2013-04-22, 15:17
+
Supun Kamburugamuva 2013-04-22, 15:27
Copy link to this message
-
Re: GSOC: Monitor Improvements
Supun Kamburugamuva 2013-04-22, 14:33
I can see how summaries are very helpful to a user. We can introduce new
fields to the existing table/tablet summery tables that displays problem
information etc.

To make the JMX polling time configurable we can introduce configuration
parameters.

For the JMX statistics we can keep data at the server for a constant time
to avoid memory growth. I think the stats are stored in memory (please
correct me if I'm wrong). If that is the case, is it possible to store them
in accumulo tables?

Thanks,
Supun...

On Mon, Apr 22, 2013 at 9:05 AM, Eric Newton <[EMAIL PROTECTED]> wrote:

> Another thing to consider is scale.  On large clusters (many hundreds of
> nodes), more data is not helpful for visualization.  Instead, summaries,
> averages and outliers are important.
>
> For example, if one node is consistently slow, it is better to know that
> than to see one graph with low numbers in a sea of graphs.
> If the monitor collects information using JMX, collection time for each
> node would be a good thing to know, too.
>
>
> -Eric
>
>
> On Sun, Apr 21, 2013 at 10:00 PM, Josh Elser <[EMAIL PROTECTED]> wrote:
>
> > Supun,
> >
> > Yup, very much so. Having a way to consume any and all metrics via JMX
> > would simplify things for any consumers (internal or external).
> >
> >
> >
> > On 04/21/2013 02:15 PM, Supun Kamburugamuva wrote:
> >
> >> Hi Josh,
> >>
> >> Thanks for the suggestions. I'll incorporate these to the proposal.
> >>
> >> Another area I would like to work is on JMX. There is a Jira that says
> to
> >> replace the Monitor calls from Thrift to JMX (Accumulo 694). Do you
> think
> >> this is a good addition to the Monitor?
> >>
> >> Thanks,
> >> Supun..
> >>
> >>
> >> On Sun, Apr 21, 2013 at 1:45 PM, Josh Elser <[EMAIL PROTECTED]>
> wrote:
> >>
> >>  Supun,
> >>>
> >>> Looks good! Can I make some suggestions/comments?
> >>>
> >>> For: "Per table plots: ACCUMULO-594", I'd also like to see minor
> >>> compactions, major compactions, index cache hit rate, and data cache
> hit
> >>> rate per table (same graphs that are displayed system-wide when you
> visit
> >>> http://${MONITOR_HOST}:50095/.
> >>>
> >>> For "Per tablet [server] plots", it would be neat if you could also
> >>> extract some general statistics like top N least performing, top N
> >>> highest
> >>> performing, etc. tablet servers. Ideally, this could correlate with
> >>> servers
> >>> that may be having problems :).
> >>>
> >>> Do you see these proposed changes as being sufficient for 3-4 months of
> >>> 40hrs/week work? If you plan to really dig into these changes (perhaps
> >>> reworking components of the monitor itself), I could perhaps see this.
> Do
> >>> you have any ideas for more lofty goals that you could pursue as well?
> I
> >>> don't want you/us to get one month into things and see you complete
> >>> everything we initially planned to accomplish :)
> >>>
> >>> - Josh
> >>>
> >>>
> >>> On 04/21/2013 10:37 AM, Supun Kamburugamuva wrote:
> >>>
> >>>  Hi all,
> >>>>
> >>>> I would like to start writing the proposal for the GSoc. I've put
> >>>> together
> >>>> some initial high level goals of the project. Please let me know what
> I
> >>>> can
> >>>> improve.
> >>>>
> >>>> Per table plots: Accumulo 594
> >>>> ---------------------
> >>>>
> >>>> The goal of this is to display plots that explains the various
> >>>> activtities
> >>>> that happens per table. When we go to the tables page of the monitor
> and
> >>>> go
> >>>> to a specific table it displays some information in a table format. We
> >>>> can
> >>>> argument this information by showing graphs for
> >>>>
> >>>> 1. Ingest entries
> >>>> 2. Ingest data size
> >>>> 3. Scan entries
> >>>> 4. Scan data size
> >>>>
> >>>> Per tablet plots
> >>>> ----------------------
> >>>>
> >>>> Same as in the table plots we can display information regarding tablet
> >>>> servers in the tablet server page. The plots will display the same
> >>>> information as table plots considering data per tablet server.

Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: [EMAIL PROTECTED];  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
+
Eric Newton 2013-04-22, 15:03
+
Gabe Bell 2013-04-22, 15:09