-Re: GSOC: Monitor improvements - draft proposal
Supun Kamburugamuva 2013-05-03, 14:01
I've submitted the proposal to google.
On Tue, Apr 30, 2013 at 10:31 AM, Supun Kamburugamuva <[EMAIL PROTECTED]>wrote:
> Hi Josh,
> Thanks for the detailed feedback. I've integrated your suggestions to the
> For a third party monitoring tool I would like to use Zabbix. But I may
> have to do more research on this one. For now I'll leave it with Zabbix.
> I couldn't find a library that helps with JMX development. I guess most
> people use the Java API directly. Anyways I'll do more research on this one
> and try to find one if possible.
> Thanks for the Javascrip library suggestions. d3.js seems impressive.
> I've updated the timeline and added a deliverable section. Hope this helps
> a bit. Let me know if it needs further improvements. For each phase my plan
> is to create a patch with the changes.
> On Mon, Apr 29, 2013 at 10:26 PM, Josh Elser <[EMAIL PROTECTED]> wrote:
>> Thanks for the draft! Some feedback -- hopefully it's useful for your
>> proposal in addition to giving you a better understanding of how Accumulo
>> is typically run.
>> "These servers perform different functionalities"
>> Actually, most serversin an Accumulo cluster are identical to one
>> another: most are running a TabletServer, and in <1.5, a Logger. The
>> exceptions are the Master, Monitor, Tracer and GarbageCollector. Master,
>> monitor and gc are typically run on the same node (monitor and gc are
>> rather lightweight). Running a tracer on every TabletServer is probably
>> overkill, but, again, this is another lightweight process, so not outside
>> the realm of possibilities.
>> "Create a JMX API for Monitor to gather statistics"
>> Any plans to include an example 3rd-party monitor that takes advantage of
>> the internal change from Thrift to JMX? If so, which? I could see this
>> being very useful for your own verification and validation, not to mention
>> for 3rd parties (people other than yourself).
>> "Table Graphs"
>> I'd be rather interested to see how the amount of data being returned by
>> a TabletServer correlates with query rate. It would be a neat plot to see
>> how RFile index size and size of each key-value returned corresponds with
>> query rate. Maybe it would be cool to have the ability to let users create
>> composite graphs?
>> "Trace Visualization"
>> Not a lot to really see here. Currently you get some rudimentary
>> information about how long it took to determine which files to delete, and
>> how long deleting them took (I think). It would be nice to see this broken
>> down by table, and include file size and other file metadata.
>> "Server Status Information"
>> I remember hearing that someone had done some work to actually pop a
>> shell in the monitor when authenticated over HTTPS. Another cool feature
>> might be to actually have some greater insight into a node (perhaps using
>> JMX calls that we wouldn't want publicly available) when properly
>> authenticated? I'm thinking about being able to view the list of running
>> scans on a node... being able to introspect the actual scan options/data,
>> ranges being run, etc.
>> "Mock Stats Collector"
>> I would put money that this will pay off in spades as you move forward
>> testing things.
>> Some more high-level things...
>> * Any thought/preference on the JMX library you would want to use?
>> and/or nvd3. Lots of options here, but licensing can be a concern. Glad you
>> thought about that already.
>> "Deliverables and Timeline"
>> I'd try to rethink your timeline a bit; it comes off very waterfall-y to
>> me. The biggest red-flag to me is the "write documentation" as your last
>> phase. Coming from experience, this doesn't work 95% of the time. Something
>> else always comes up, takes longer, w/e and suddenly you have some code
Member, Apache Software Foundation; http://www.apache.org
E-mail: [EMAIL PROTECTED]; Mobile: +1 812 369 6762