Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> RE: How Yarn execute MRv1 job?


+
Devaraj k 2013-06-19, 05:35
+
Rahul Bhattacharjee 2013-06-19, 05:41
+
Arun C Murthy 2013-06-19, 05:54
+
Rahul Bhattacharjee 2013-06-19, 10:20
+
sam liu 2013-06-20, 01:45
+
Arun C Murthy 2013-06-20, 04:12
+
sam liu 2013-06-20, 06:11
+
Azuryy Yu 2013-06-20, 06:33
+
sam liu 2013-06-20, 06:56
+
Arun C Murthy 2013-06-20, 06:59
+
Azuryy Yu 2013-06-20, 07:17
Copy link to this message
-
Re: How Yarn execute MRv1 job?
Got it, and thanks!
2013/6/20 Azuryy Yu <[EMAIL PROTECTED]>

> HBase-0.94.* does support hadoop-2.x, do you look at the web site i
> provided?
>
> Hive-0.9.0  doesn't  support hadoop-2.x
>
>
>
>
> On Thu, Jun 20, 2013 at 2:59 PM, Arun C Murthy <[EMAIL PROTECTED]>wrote:
>
>> I'd use hive-0.11.
>>
>> On Jun 19, 2013, at 11:56 PM, sam liu <[EMAIL PROTECTED]> wrote:
>>
>> Hi Azurry,
>>
>> So, older versions of HBase and Hive, like HBase 0.94.0 and Hive 0.9.0,
>> does not support hadoop 2.x, right?
>>
>> Thanks!
>>
>>
>> 2013/6/20 Azuryy Yu <[EMAIL PROTECTED]>
>>
>>> Hi Sam,
>>> please look at :http://hbase.apache.org/book.html#d2617e499
>>>
>>> generally, we said YARN is Hadoop-2.x, you can download
>>> hadoop-2.0.4-alpha. and Hive-0.10 supports hadoop-2.x very well.
>>>
>>>
>>>
>>> On Thu, Jun 20, 2013 at 2:11 PM, sam liu <[EMAIL PROTECTED]> wrote:
>>>
>>>> Thanks Arun!
>>>>
>>>> #1, Yes, I did tests and found that the MRv1 jobs could run against
>>>> YARN directly, without recompiling
>>>>
>>>> #2, do you mean the old versions of HBase/Hive can not run agains YARN,
>>>> and only some special versions of them can run against YARN? If yes, how
>>>> can I get the versions for YARN?
>>>>
>>>>
>>>> 2013/6/20 Arun C Murthy <[EMAIL PROTECTED]>
>>>>
>>>>>
>>>>> On Jun 19, 2013, at 6:45 PM, sam liu <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>> Appreciating for the detailed answers! Here are three further
>>>>> questions:
>>>>>
>>>>> - Yarn maintains backwards compatibility, and MRv1 job could run on
>>>>> Yarn. If yarn does not ask existing MRv1 job to do any code change, but why
>>>>> we should recompile the MRv1 job?
>>>>>
>>>>>
>>>>> You don't need to recompile MRv1 jobs to run against YARN.
>>>>>
>>>>> - Which yarn jar files are required in the recompiling?
>>>>> - In a cluster with Hadoop 1.1.1 and other Hadoop related
>>>>> components(HBase 0.94.3,  Hive 0.9.0, Zookeeper 3.4.5,...), if we want to
>>>>> replace Hadoop 1.1.1 with yarn, do we need to recompile all other Hadoop
>>>>> related components again with yarn jar files? Without any code change?
>>>>>
>>>>>
>>>>> You will need versions of HBase, Hive etc. which are integrated with
>>>>> hadoop-2.x, but not need to change any of your end-user applications (MR
>>>>> jobs, hive queries, pig scripts etc.)
>>>>>
>>>>> Arun
>>>>>
>>>>>
>>>>> Thanks in advance!
>>>>>
>>>>>
>>>>>
>>>>> 2013/6/19 Rahul Bhattacharjee <[EMAIL PROTECTED]>
>>>>>
>>>>>> Thanks Arun and Devraj , good to know.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 19, 2013 at 11:24 AM, Arun C Murthy <[EMAIL PROTECTED]>wrote:
>>>>>>
>>>>>>> Not true, the CapacityScheduler has support for both CPU & Memory
>>>>>>> now.
>>>>>>>
>>>>>>> On Jun 18, 2013, at 10:41 PM, Rahul Bhattacharjee <
>>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>>
>>>>>>> Hi Devaraj,
>>>>>>>
>>>>>>> As for the container request request for yarn container , currently
>>>>>>> only memory is considered as resource , not cpu. Please correct.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Rahul
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 19, 2013 at 11:05 AM, Devaraj k <[EMAIL PROTECTED]>wrote:
>>>>>>>
>>>>>>>>  Hi Sam,****
>>>>>>>>
>>>>>>>>   Please find the answers for your queries. ****
>>>>>>>>
>>>>>>>>
>>>>>>>> >- Yarn could run multiple kinds of jobs(MR, MPI, ...), but, MRv1
>>>>>>>> job has special execution process(map > shuffle > reduce) in Hadoop 1.x,
>>>>>>>> and how Yarn execute a MRv1 job? still include some special MR steps in
>>>>>>>> Hadoop 1.x, like map, sort, merge, combine and shuffle?****
>>>>>>>>
>>>>>>>> ** **
>>>>>>>>
>>>>>>>> In Yarn, it is a concept of application. MR Job is one kind of
>>>>>>>> application which makes use of MRAppMaster(i.e ApplicationMaster for the
>>>>>>>> application). If we want to run different kinds of applications we should
>>>>>>>> have ApplicationMaster for each kind of application.****
>>>>>>>>
>>>>>>>> ** **
>>>>>>>>
>>>>>>>> >- Do the MRv1 parameters still work for Yarn? Like
>>>>>>>> mapreduce.task.io.sort.mb and mapreduce.map.sort.spill.percent?****
+
Rahul Bhattacharjee 2013-06-19, 05:42
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB