Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # dev >> Optimized Hadoop


+
Anty 2012-02-16, 15:01
+
Arun C Murthy 2012-02-16, 15:38
+
brisk 2012-02-28, 21:08
+
Schubert Zhang 2012-02-16, 17:44
+
Todd Lipcon 2012-02-16, 19:27
+
Anty 2012-02-17, 03:00
+
Schubert Zhang 2012-02-17, 04:25
Copy link to this message
-
Re: Optimized Hadoop
On Thu, Feb 16, 2012 at 8:25 PM, Schubert Zhang <[EMAIL PROTECTED]> wrote:
> 1) it should be sort-avoidance.

right - that's a nice improvement, looking forward to getting that in
trunk at some point.

> 2) work pool (like Tenzing)
>

Looking at the code, it seems you only support the default task
executor. Do you have plans to support run-as-user through the linux
task-controller? It's a requirement for secure environments. But, it
makes the worker pool model a little tougher since you can't share a
JVM cross-user.

Also, how does class-unloading and reloading interact with this model?

> Sorry ,the adaptive heartbeat code is not in this github code, we are
> discussing it.
>
>
>
> On Fri, Feb 17, 2012 at 11:00 AM, Anty <[EMAIL PROTECTED]> wrote:
>>
>> Hi: Todd
>>
>> yes, the rewritten shuffle in actual a backport of the shuffle from MR2 .
>> We mainly add the following two features:
>> 1) shuffle avoidance
>> 2) work pool
>>
>>
>> On Fri, Feb 17, 2012 at 3:27 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>>
>>> Hey Schubert,
>>>
>>> Looking at the code on github, it looks like your rewritten shuffle is
>>> in fact just a backport of the shuffle from MR2. I didn't look closely
>>> - are there any distinguishing factors?
>>> Also, the OOB heartbeat and adaptive heartbeat code seems to be the
>>> same as what's in 1.0?
>>>
>>> -Todd
>>>
>>> On Thu, Feb 16, 2012 at 9:44 AM, Schubert Zhang <[EMAIL PROTECTED]>
>>> wrote:
>>> > Here is the presentation to describe our job,
>>> >
>>> > http://www.slideshare.net/hanborq/hanborq-optimizations-on-hadoop-mapreduce-20120216a
>>> > Wellcome to give your advises.
>>> > It's just a little step, and we are continue to do more improvements,
>>> > thanks
>>> > for your help.
>>> >
>>> >
>>> >
>>> >
>>> > On Thu, Feb 16, 2012 at 11:01 PM, Anty <[EMAIL PROTECTED]> wrote:
>>> >>
>>> >> Hi: Guys
>>> >>        We just deliver a optimized hadoop , if you are interested, Pls
>>> >> refer to https://github.com/hanborq/hadoop
>>> >>
>>> >> --
>>> >> Best Regards
>>> >> Anty Rao
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>
>>
>>
>>
>> --
>> Best Regards
>> Anty Rao
>
>

--
Todd Lipcon
Software Engineer, Cloudera
+
Anty 2012-02-18, 15:12
+
Schubert Zhang 2012-02-20, 18:17
+
Sharad Agarwal 2012-02-17, 11:18
+
Schubert Zhang 2012-02-23, 10:49
+
Schubert Zhang 2012-02-23, 11:01
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB