Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> How to write/run MPI program on Yarn?


Copy link to this message
-
Re: How to write/run MPI program on Yarn?
Sam,

The fundamental idea of YARN is to split up the two major functionalities of
the JobTracker, resource management and job scheduling/monitoring, into
separate daemons. With YARN, we're going to be able to run multiple
workload, one of them being MapReduce another one being MPI. So mpich2-yarn
is the port of MPI to run on top of yarn.

You may want to read :
http://hortonworks.com/hadoop/yarn/
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html

Hope it helps,
Olivier
On 2 July 2013 02:21, sam liu <[EMAIL PROTECTED]> wrote:

> Any one could help answer above questions? Thanks a lot!
>
>
> 2013/7/1 sam liu <[EMAIL PROTECTED]>
>
>> Thanks Pramod and Clark!
>>
>> 1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
>> 2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
>> there is already a JIRA:
>> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>>
>>
>> 2013/7/1 Clark Yang (杨卓荦) <[EMAIL PROTECTED]>
>>
>> Hi, sam
>>>
>>>    Please try it following the README.md on
>>> https://github.com/clarkyzl/mpich2-yarn/.
>>>   The design is simple ,make each nodemanger runs a "smpd" daemon and
>>> make application master run "mpiexec" to submit mpi jobs. The biggest
>>> problem here, I think, is the yarn scheduler do not support gang
>>> secheduling algorithm, and the scheduling is very very slow.
>>>
>>>   Any ideas please feel free to share.
>>>
>>> Cheers,
>>> Zhuoluo (Clark) Yang
>>>
>>>
>>> 2013/6/30 Pramod N <[EMAIL PROTECTED]>
>>>
>>>> Hi Sam,
>>>> This might interest you.
>>>> https://github.com/clarkyzl/mpich2-yarn
>>>>
>>>>
>>>>
>>>> Pramod N <http://atmachinelearner.blogspot.in>
>>>> Bruce Wayne of web
>>>> @machinelearner <https://twitter.com/machinelearner>
>>>>
>>>> --
>>>>
>>>>
>>>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Hi Experts,
>>>>>
>>>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>>>
>>>>> - If yes, is there any example of writing a MPI program on Yarn? How
>>>>> to write the client side code to configure and submit the MPI job?
>>>>> - If no, which Hadoop version will support MPI programming?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>
>>
>
--
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
[EMAIL PROTECTED]
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB