Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - Notes from yesterday's Contributors Pow Wow [WAS --> Re: HBase Developer's Pow-wow]


Copy link to this message
-
Re: Notes from yesterday's Contributors Pow Wow [WAS --> Re: HBase Developer's Pow-wow]
Enis Söztutar 2012-09-12, 18:30
Thanks Stack for the notes, and thanks everyone for coming.

Cheers,
Enis

On Wed, Sep 12, 2012 at 11:21 AM, Stack <[EMAIL PROTECTED]> wrote:

> Here are notes from yesterday's contributor's meetup:
> http://www.meetup.com/hbaseusergroup/events/80621872/
>
> The notes below are spotty.  I kept forgetting to take them.  I also
> was unable to keep up w/ the rate of exchange so in many parts the
> reporting drops or a speaker's nuanced argument is crassly rendered.
>
> We'll hire a stenographer for the next one.
>
> We started at 2:00PM.  Meeting lasted till almost 6:00PM
>
> Devaraj Das of HortonWorks, our host, welcomed everyone.
>
> Jimmy Xiang presented on recent changes to Assignment Manager in
> Master: http://files.meetup.com/1350427/assignment-manager.pdf
>
> Good Q&A during the presentation and after included:
>
> + What kind of tests do we have in place for the new AM patches?
> + We should have tests to ensure we don't lose performance
> + Make sure we don't lose operator facility; e.g. abiilty to override
> assignment state from shell
>
> A general question was posed on what do we all think of the current
> state of the Assignment Manger?  Could we make it pluggable (Francis
> Liu is looking at making the AM pluggable so can add "Groups")?  Will
> we ever be able to make the AM rock solid?
>
> Discussion followed
>
> Its complex.  Its hard making it pluggable?  Or how about adding
> support for different kinds of policies?
>
> Jon Hsieh: Rules that were there originally in design, are they being
> followed currently?
>
> Elliott: AM stuff bled into the Master; stuff is bleeding all over.
> Need to clean up Master.
> Enis: Splitting state is split between zk, RS, and HM.... too complex.
>  Could AM/Master just do it? Some one entity should be the source of
> truth [for region assignment].
> Lars Hofhansl: We need to write out the state machine and just rewrite
> the AM or hack code to get same result
> Andrew Purtell: We should do both.  Testable changes.  We have gone
> through a bunch of master rewrites and suspect that  another rewrite
> would just land us with a new set of issues.
> Lars: If we had AM state machine, then could [do both rewrite and/or
> patch it to a state of robustness].
> Jon Hsieh: Is Master2 close to its original design?  Maybe we should
> do assignment in another way?  Maybe Ram[krishna] understands the AM
> but it seems like no one else here really does.  Yeah, Ram is the AM
> (he can assign to us how to fix it all).
> Ted Yu: AM should be able to do colocation for secondary indices and
> region grouping.
> Francis Liu: On what region grouping is, if could do different AM,
> could then assign tables to a group, could make it so they don't
> affect another application running in a different group all on a big
> cluster.  Same for workloads.  Were thinking of doing grouping first,
> then attack multi-tenancy later.
> LarsH: Do you need the whole thing pluggable or do you want to rewrite it.
> Francis: Pluggable would be nice.  In the past  have subclassed AM to
> add functionality (would like to avoid that).
> JDCryans: Could you build grouping on top of HBase by just disabling
> the balancer and use the move command?
> Elliott: Could you do your own balancer?  Would that work?
> Francis: Balancer is given a plan only
> Andrew: Why not do as Karthik suggested in the past, and just run
> mutiple hbases?
> Francis: Its too complicated
> Ted: (Said something about explaining Francis's situation)
> Jacques: Seems like you want a placement strategy only?  Or do you
> need to change timeouts, etc?  Or is it just placement?
> Elliott: Would it work if we added more facility to the balancer?
> Maybe make it more pluggable with more levers?
> Andrew: Yeah, balancer might be way to go for 2ndary indices because
> want to colocate regions.
>
> Enis presented on the new Integration Test patch:
> http://files.meetup.com/1350427/HBase-meetup-HW.pptx
>
> Andrew: Intends to use IT internally. Is looking at porting some of