Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - How to write/run MPI program on Yarn?


Copy link to this message
-
Re: How to write/run MPI program on Yarn?
Olivier Renault 2013-07-02, 09:29
Sam,

The fundamental idea of YARN is to split up the two major functionalities of
the JobTracker, resource management and job scheduling/monitoring, into
separate daemons. With YARN, we're going to be able to run multiple
workload, one of them being MapReduce another one being MPI. So mpich2-yarn
is the port of MPI to run on top of yarn.

You may want to read :
http://hortonworks.com/hadoop/yarn/
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html

Hope it helps,
Olivier
On 2 July 2013 02:21, sam liu <[EMAIL PROTECTED]> wrote:

> Any one could help answer above questions? Thanks a lot!
>
>
> 2013/7/1 sam liu <[EMAIL PROTECTED]>
>
>> Thanks Pramod and Clark!
>>
>> 1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
>> 2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
>> there is already a JIRA:
>> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>>
>>
>> 2013/7/1 Clark Yang (杨卓荦) <[EMAIL PROTECTED]>
>>
>> Hi, sam
>>>
>>>    Please try it following the README.md on
>>> https://github.com/clarkyzl/mpich2-yarn/.
>>>   The design is simple ,make each nodemanger runs a "smpd" daemon and
>>> make application master run "mpiexec" to submit mpi jobs. The biggest
>>> problem here, I think, is the yarn scheduler do not support gang
>>> secheduling algorithm, and the scheduling is very very slow.
>>>
>>>   Any ideas please feel free to share.
>>>
>>> Cheers,
>>> Zhuoluo (Clark) Yang
>>>
>>>
>>> 2013/6/30 Pramod N <[EMAIL PROTECTED]>
>>>
>>>> Hi Sam,
>>>> This might interest you.
>>>> https://github.com/clarkyzl/mpich2-yarn
>>>>
>>>>
>>>>
>>>> Pramod N <http://atmachinelearner.blogspot.in>
>>>> Bruce Wayne of web
>>>> @machinelearner <https://twitter.com/machinelearner>
>>>>
>>>> --
>>>>
>>>>
>>>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Hi Experts,
>>>>>
>>>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>>>
>>>>> - If yes, is there any example of writing a MPI program on Yarn? How
>>>>> to write the client side code to configure and submit the MPI job?
>>>>> - If no, which Hadoop version will support MPI programming?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>
>>
>
--
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
[EMAIL PROTECTED]
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>