Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> Optimized Hadoop


Copy link to this message
-
Re: Optimized Hadoop
On Thu, Feb 16, 2012 at 8:25 PM, Schubert Zhang <[EMAIL PROTECTED]> wrote:
> 1) it should be sort-avoidance.

right - that's a nice improvement, looking forward to getting that in
trunk at some point.

> 2) work pool (like Tenzing)
>

Looking at the code, it seems you only support the default task
executor. Do you have plans to support run-as-user through the linux
task-controller? It's a requirement for secure environments. But, it
makes the worker pool model a little tougher since you can't share a
JVM cross-user.

Also, how does class-unloading and reloading interact with this model?

> Sorry ,the adaptive heartbeat code is not in this github code, we are
> discussing it.
>
>
>
> On Fri, Feb 17, 2012 at 11:00 AM, Anty <[EMAIL PROTECTED]> wrote:
>>
>> Hi: Todd
>>
>> yes, the rewritten shuffle in actual a backport of the shuffle from MR2 .
>> We mainly add the following two features:
>> 1) shuffle avoidance
>> 2) work pool
>>
>>
>> On Fri, Feb 17, 2012 at 3:27 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>>
>>> Hey Schubert,
>>>
>>> Looking at the code on github, it looks like your rewritten shuffle is
>>> in fact just a backport of the shuffle from MR2. I didn't look closely
>>> - are there any distinguishing factors?
>>> Also, the OOB heartbeat and adaptive heartbeat code seems to be the
>>> same as what's in 1.0?
>>>
>>> -Todd
>>>
>>> On Thu, Feb 16, 2012 at 9:44 AM, Schubert Zhang <[EMAIL PROTECTED]>
>>> wrote:
>>> > Here is the presentation to describe our job,
>>> >
>>> > http://www.slideshare.net/hanborq/hanborq-optimizations-on-hadoop-mapreduce-20120216a
>>> > Wellcome to give your advises.
>>> > It's just a little step, and we are continue to do more improvements,
>>> > thanks
>>> > for your help.
>>> >
>>> >
>>> >
>>> >
>>> > On Thu, Feb 16, 2012 at 11:01 PM, Anty <[EMAIL PROTECTED]> wrote:
>>> >>
>>> >> Hi: Guys
>>> >>        We just deliver a optimized hadoop , if you are interested, Pls
>>> >> refer to https://github.com/hanborq/hadoop
>>> >>
>>> >> --
>>> >> Best Regards
>>> >> Anty Rao
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>
>>
>>
>>
>> --
>> Best Regards
>> Anty Rao
>
>

--
Todd Lipcon
Software Engineer, Cloudera