-Re: Notes from yesterday's Contributors Pow Wow [WAS --> Re: HBase Developer's Pow-wow]
Thanks Stack for the notes, and thanks everyone for coming.
On Wed, Sep 12, 2012 at 11:21 AM, Stack <[EMAIL PROTECTED]> wrote:
> Here are notes from yesterday's contributor's meetup:
> The notes below are spotty. I kept forgetting to take them. I also
> was unable to keep up w/ the rate of exchange so in many parts the
> reporting drops or a speaker's nuanced argument is crassly rendered.
> We'll hire a stenographer for the next one.
> We started at 2:00PM. Meeting lasted till almost 6:00PM
> Devaraj Das of HortonWorks, our host, welcomed everyone.
> Jimmy Xiang presented on recent changes to Assignment Manager in
> Master: http://files.meetup.com/1350427/assignment-manager.pdf
> Good Q&A during the presentation and after included:
> + What kind of tests do we have in place for the new AM patches?
> + We should have tests to ensure we don't lose performance
> + Make sure we don't lose operator facility; e.g. abiilty to override
> assignment state from shell
> A general question was posed on what do we all think of the current
> state of the Assignment Manger? Could we make it pluggable (Francis
> Liu is looking at making the AM pluggable so can add "Groups")? Will
> we ever be able to make the AM rock solid?
> Discussion followed
> Its complex. Its hard making it pluggable? Or how about adding
> support for different kinds of policies?
> Jon Hsieh: Rules that were there originally in design, are they being
> followed currently?
> Elliott: AM stuff bled into the Master; stuff is bleeding all over.
> Need to clean up Master.
> Enis: Splitting state is split between zk, RS, and HM.... too complex.
> Could AM/Master just do it? Some one entity should be the source of
> truth [for region assignment].
> Lars Hofhansl: We need to write out the state machine and just rewrite
> the AM or hack code to get same result
> Andrew Purtell: We should do both. Testable changes. We have gone
> through a bunch of master rewrites and suspect that another rewrite
> would just land us with a new set of issues.
> Lars: If we had AM state machine, then could [do both rewrite and/or
> patch it to a state of robustness].
> Jon Hsieh: Is Master2 close to its original design? Maybe we should
> do assignment in another way? Maybe Ram[krishna] understands the AM
> but it seems like no one else here really does. Yeah, Ram is the AM
> (he can assign to us how to fix it all).
> Ted Yu: AM should be able to do colocation for secondary indices and
> region grouping.
> Francis Liu: On what region grouping is, if could do different AM,
> could then assign tables to a group, could make it so they don't
> affect another application running in a different group all on a big
> cluster. Same for workloads. Were thinking of doing grouping first,
> then attack multi-tenancy later.
> LarsH: Do you need the whole thing pluggable or do you want to rewrite it.
> Francis: Pluggable would be nice. In the past have subclassed AM to
> add functionality (would like to avoid that).
> JDCryans: Could you build grouping on top of HBase by just disabling
> the balancer and use the move command?
> Elliott: Could you do your own balancer? Would that work?
> Francis: Balancer is given a plan only
> Andrew: Why not do as Karthik suggested in the past, and just run
> mutiple hbases?
> Francis: Its too complicated
> Ted: (Said something about explaining Francis's situation)
> Jacques: Seems like you want a placement strategy only? Or do you
> need to change timeouts, etc? Or is it just placement?
> Elliott: Would it work if we added more facility to the balancer?
> Maybe make it more pluggable with more levers?
> Andrew: Yeah, balancer might be way to go for 2ndary indices because
> want to colocate regions.
> Enis presented on the new Integration Test patch:
> Andrew: Intends to use IT internally. Is looking at porting some of