Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo, mail # dev - GSOC: Monitor Improvements


Copy link to this message
-
Re: GSOC: Monitor Improvements
David Medinets 2013-04-22, 14:41
The average developer probably can't access a large cluster with hundred of
nodes. Is there a way to simulate this?
On Mon, Apr 22, 2013 at 9:05 AM, Eric Newton <[EMAIL PROTECTED]> wrote:

> Another thing to consider is scale.  On large clusters (many hundreds of
> nodes), more data is not helpful for visualization.  Instead, summaries,
> averages and outliers are important.
>
> For example, if one node is consistently slow, it is better to know that
> than to see one graph with low numbers in a sea of graphs.
>
> If the monitor collects information using JMX, collection time for each
> node would be a good thing to know, too.
>
> -Eric
>
>
> On Sun, Apr 21, 2013 at 10:00 PM, Josh Elser <[EMAIL PROTECTED]> wrote:
>
> > Supun,
> >
> > Yup, very much so. Having a way to consume any and all metrics via JMX
> > would simplify things for any consumers (internal or external).
> >
> >
> >
> > On 04/21/2013 02:15 PM, Supun Kamburugamuva wrote:
> >
> >> Hi Josh,
> >>
> >> Thanks for the suggestions. I'll incorporate these to the proposal.
> >>
> >> Another area I would like to work is on JMX. There is a Jira that says
> to
> >> replace the Monitor calls from Thrift to JMX (Accumulo 694). Do you
> think
> >> this is a good addition to the Monitor?
> >>
> >> Thanks,
> >> Supun..
> >>
> >>
> >> On Sun, Apr 21, 2013 at 1:45 PM, Josh Elser <[EMAIL PROTECTED]>
> wrote:
> >>
> >>  Supun,
> >>>
> >>> Looks good! Can I make some suggestions/comments?
> >>>
> >>> For: "Per table plots: ACCUMULO-594", I'd also like to see minor
> >>> compactions, major compactions, index cache hit rate, and data cache
> hit
> >>> rate per table (same graphs that are displayed system-wide when you
> visit
> >>> http://${MONITOR_HOST}:50095/.
> >>>
> >>> For "Per tablet [server] plots", it would be neat if you could also
> >>> extract some general statistics like top N least performing, top N
> >>> highest
> >>> performing, etc. tablet servers. Ideally, this could correlate with
> >>> servers
> >>> that may be having problems :).
> >>>
> >>> Do you see these proposed changes as being sufficient for 3-4 months of
> >>> 40hrs/week work? If you plan to really dig into these changes (perhaps
> >>> reworking components of the monitor itself), I could perhaps see this.
> Do
> >>> you have any ideas for more lofty goals that you could pursue as well?
> I
> >>> don't want you/us to get one month into things and see you complete
> >>> everything we initially planned to accomplish :)
> >>>
> >>> - Josh
> >>>
> >>>
> >>> On 04/21/2013 10:37 AM, Supun Kamburugamuva wrote:
> >>>
> >>>  Hi all,
> >>>>
> >>>> I would like to start writing the proposal for the GSoc. I've put
> >>>> together
> >>>> some initial high level goals of the project. Please let me know what
> I
> >>>> can
> >>>> improve.
> >>>>
> >>>> Per table plots: Accumulo 594
> >>>> ---------------------
> >>>>
> >>>> The goal of this is to display plots that explains the various
> >>>> activtities
> >>>> that happens per table. When we go to the tables page of the monitor
> and
> >>>> go
> >>>> to a specific table it displays some information in a table format. We
> >>>> can
> >>>> argument this information by showing graphs for
> >>>>
> >>>> 1. Ingest entries
> >>>> 2. Ingest data size
> >>>> 3. Scan entries
> >>>> 4. Scan data size
> >>>>
> >>>> Per tablet plots
> >>>> ----------------------
> >>>>
> >>>> Same as in the table plots we can display information regarding tablet
> >>>> servers in the tablet server page. The plots will display the same
> >>>> information as table plots considering data per tablet server.
> >>>>
> >>>> Trace Visualization: Accumulo 1198
> >>>> ----------------------------
> >>>>
> >>>> Since we are displaying graphs about each tablet and each table we can
> >>>> add
> >>>> major and minor compaction graph to each table and each tablet.
> >>>>
> >>>> Or other option is to display this in a single graph in overview page
> >>>> with
> >>>> different graph lines for different tables and tablets.