Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Notes from yesterday's Contributors Pow Wow [WAS --> Re: HBase Developer's Pow-wow]


Copy link to this message
-
RE: Notes from yesterday's Contributors Pow Wow [WAS --> Re: HBase Developer's Pow-wow]
Thanks for the write up Stack.  Some of my comments I can add up in a new
thread?

Regards
Ram

> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of
> Stack
> Sent: Wednesday, September 12, 2012 11:51 PM
> To: [EMAIL PROTECTED]
> Subject: Notes from yesterday's Contributors Pow Wow [WAS --> Re: HBase
> Developer's Pow-wow]
>
> Here are notes from yesterday's contributor's meetup:
> http://www.meetup.com/hbaseusergroup/events/80621872/
>
> The notes below are spotty.  I kept forgetting to take them.  I also
> was unable to keep up w/ the rate of exchange so in many parts the
> reporting drops or a speaker's nuanced argument is crassly rendered.
>
> We'll hire a stenographer for the next one.
>
> We started at 2:00PM.  Meeting lasted till almost 6:00PM
>
> Devaraj Das of HortonWorks, our host, welcomed everyone.
>
> Jimmy Xiang presented on recent changes to Assignment Manager in
> Master: http://files.meetup.com/1350427/assignment-manager.pdf
>
> Good Q&A during the presentation and after included:
>
> + What kind of tests do we have in place for the new AM patches?
> + We should have tests to ensure we don't lose performance
> + Make sure we don't lose operator facility; e.g. abiilty to override
> assignment state from shell
>
> A general question was posed on what do we all think of the current
> state of the Assignment Manger?  Could we make it pluggable (Francis
> Liu is looking at making the AM pluggable so can add "Groups")?  Will
> we ever be able to make the AM rock solid?
>
> Discussion followed
>
> Its complex.  Its hard making it pluggable?  Or how about adding
> support for different kinds of policies?
>
> Jon Hsieh: Rules that were there originally in design, are they being
> followed currently?
>
> Elliott: AM stuff bled into the Master; stuff is bleeding all over.
> Need to clean up Master.
> Enis: Splitting state is split between zk, RS, and HM.... too complex.
>  Could AM/Master just do it? Some one entity should be the source of
> truth [for region assignment].
> Lars Hofhansl: We need to write out the state machine and just rewrite
> the AM or hack code to get same result
> Andrew Purtell: We should do both.  Testable changes.  We have gone
> through a bunch of master rewrites and suspect that  another rewrite
> would just land us with a new set of issues.
> Lars: If we had AM state machine, then could [do both rewrite and/or
> patch it to a state of robustness].
> Jon Hsieh: Is Master2 close to its original design?  Maybe we should
> do assignment in another way?  Maybe Ram[krishna] understands the AM
> but it seems like no one else here really does.  Yeah, Ram is the AM
> (he can assign to us how to fix it all).
> Ted Yu: AM should be able to do colocation for secondary indices and
> region grouping.
> Francis Liu: On what region grouping is, if could do different AM,
> could then assign tables to a group, could make it so they don't
> affect another application running in a different group all on a big
> cluster.  Same for workloads.  Were thinking of doing grouping first,
> then attack multi-tenancy later.
> LarsH: Do you need the whole thing pluggable or do you want to rewrite
> it.
> Francis: Pluggable would be nice.  In the past  have subclassed AM to
> add functionality (would like to avoid that).
> JDCryans: Could you build grouping on top of HBase by just disabling
> the balancer and use the move command?
> Elliott: Could you do your own balancer?  Would that work?
> Francis: Balancer is given a plan only
> Andrew: Why not do as Karthik suggested in the past, and just run
> mutiple hbases?
> Francis: Its too complicated
> Ted: (Said something about explaining Francis's situation)
> Jacques: Seems like you want a placement strategy only?  Or do you
> need to change timeouts, etc?  Or is it just placement?
> Elliott: Would it work if we added more facility to the balancer?
> Maybe make it more pluggable with more levers?
> Andrew: Yeah, balancer might be way to go for 2ndary indices because
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB