To all,

I have a few questions regarding YARN (with respect to Hadoop):

Are YARN and Hadoop separate, or is YARN the successor to Hadoop?

What are the major conceptual differences between YARN and Hadoop?

I have further questions, but they may or may not make sense depending
on the answers to the above.

Thanks in advance!

Tom Brown

They are not comparable.

YARN also known as MRv2 is the newer version of MapReduce also known as MRv1.
-----Original Message-----
From: Tom Brown [mailto:[EMAIL PROTECTED]]
Sent: Thursday, October 18, 2012 4:33 PM
To: [EMAIL PROTECTED]
Subject: Differences between YARN and Hadoop

To all,

I have a few questions regarding YARN (with respect to Hadoop):

Are YARN and Hadoop separate, or is YARN the successor to Hadoop?

What are the major conceptual differences between YARN and Hadoop?

I have further questions, but they may or may not make sense depending on the answers to the above.

Thanks in advance!

Tom Brown
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

Hi
>From the beginning YARN was a part of the Hadoop project. Now it has
been decided to move it to a top-level apache project.
So, now there are two ways of running hadoop jobs:
1) "Old" hadoop (the current stable thing)
2) YARN + MR implementation (the future thing, still not stable AFAIK)

Hope that helps

On Fri, Oct 19, 2012 at 12:33 AM, Tom Brown <[EMAIL PROTECTED]> wrote:
> To all,
>
> I have a few questions regarding YARN (with respect to Hadoop):
>
> Are YARN and Hadoop separate, or is YARN the successor to Hadoop?
>
> What are the major conceptual differences between YARN and Hadoop?
>
> I have further questions, but they may or may not make sense depending
> on the answers to the above.
>
> Thanks in advance!
>
> Tom Brown

Right now Hadoop seems to refer to the bundle of the MRv1 framework
and HDFS though both can be used independently. Since MRv2 is its own
top level project, where does leave HDFS?

--Tom

On Thu, Oct 18, 2012 at 5:01 PM, Ruslan Al-Fakikh <[EMAIL PROTECTED]> wrote:
> Hi
> From the beginning YARN was a part of the Hadoop project. Now it has
> been decided to move it to a top-level apache project.
> So, now there are two ways of running hadoop jobs:
> 1) "Old" hadoop (the current stable thing)
> 2) YARN + MR implementation (the future thing, still not stable AFAIK)
>
> Hope that helps
>
> On Fri, Oct 19, 2012 at 12:33 AM, Tom Brown <[EMAIL PROTECTED]> wrote:
>> To all,
>>
>> I have a few questions regarding YARN (with respect to Hadoop):
>>
>> Are YARN and Hadoop separate, or is YARN the successor to Hadoop?
>>
>> What are the major conceptual differences between YARN and Hadoop?
>>
>> I have further questions, but they may or may not make sense depending
>> on the answers to the above.
>>
>> Thanks in advance!
>>
>> Tom Brown

Andy,

"YARN" is NOT "MRv2". That seems to be a major confusion with people
thanks to everyone mixing up the terms. See this small post for a
clear difference:
http://www.cloudera.com/blog/2012/10/mr2-and-yarn-briefly-explained/

On Fri, Oct 19, 2012 at 3:30 AM, Kartashov, Andy <[EMAIL PROTECTED]> wrote:
> They are not comparable.
>
> YARN also known as MRv2 is the newer version of MapReduce also known as MRv1.
>
>
>
>
> -----Original Message-----
> From: Tom Brown [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, October 18, 2012 4:33 PM
> To: [EMAIL PROTECTED]
> Subject: Differences between YARN and Hadoop
>
> To all,
>
> I have a few questions regarding YARN (with respect to Hadoop):
>
> Are YARN and Hadoop separate, or is YARN the successor to Hadoop?
>
> What are the major conceptual differences between YARN and Hadoop?
>
> I have further questions, but they may or may not make sense depending on the answers to the above.
>
> Thanks in advance!
>
> Tom Brown
> NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

--
Harsh J

Hi Tom,

On Fri, Oct 19, 2012 at 2:03 AM, Tom Brown <[EMAIL PROTECTED]> wrote:
> To all,
>
> I have a few questions regarding YARN (with respect to Hadoop):
>
> Are YARN and Hadoop separate, or is YARN the successor to Hadoop?

YARN replaces what used to be plain MapReduce, in order to provide an
API for, and to support development of applications that wish to use a
different distributed processing method than just MR, on top of HDFS.

YARN (called Apache Hadoop YARN presently) is part of Apache Hadoop.
YARN is not "Hadoop".

> What are the major conceptual differences between YARN and Hadoop?

The two can't be compared this way, see above and my previous post to Andy.

--
Harsh J

于 2012/10/19 13:33, Harsh J 写道:
> Andy,
>
> "YARN" is NOT "MRv2". That seems to be a major confusion with people
> thanks to everyone mixing up the terms. See this small post for a
> clear difference:
> http://www.cloudera.com/blog/2012/10/mr2-and-yarn-briefly-explained/
>
> On Fri, Oct 19, 2012 at 3:30 AM, Kartashov, Andy <[EMAIL PROTECTED]> wrote:
>> They are not comparable.
>>
>> YARN also known as MRv2 is the newer version of MapReduce also known as MRv1.
>>
>>
>>
>>
>> -----Original Message-----
>> From: Tom Brown [mailto:[EMAIL PROTECTED]]
>> Sent: Thursday, October 18, 2012 4:33 PM
>> To: [EMAIL PROTECTED]
>> Subject: Differences between YARN and Hadoop
>>
>> To all,
>>
>> I have a few questions regarding YARN (with respect to Hadoop):
>>
>> Are YARN and Hadoop separate, or is YARN the successor to Hadoop?
>>
>> What are the major conceptual differences between YARN and Hadoop?
>>
>> I have further questions, but they may or may not make sense depending on the answers to the above.
>>
>> Thanks in advance!
>>
>> Tom Brown
>> NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
>
>
unsubscribe

First, see what a TLP means, here:
http://incubator.apache.org/guides/graduation.html#introduction.

Apache Hadoop is a collection of 4 major sub-projects (Common, HDFS,
YARN, and MR) presently. The top level directory structure can be
found at the SVN location:
http://svn.apache.org/repos/asf/hadoop/common/trunk/

MRv2 is a sub-project of Apache Hadoop, not a TLP. MRv2 resides under
Hadoop, at http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce-project/.

YARN is also still a sub-project of Apache Hadoop, not a TLP yet. YARN
resides under Hadoop, at
http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/

HDFS is also a sub-project of Apache Hadoop, not a TLP. HDFS resides
under Hadoop, at
http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-hdfs-project/

Hope this clears your confusion :)

On Fri, Oct 19, 2012 at 3:58 AM, Tom Brown <[EMAIL PROTECTED]> wrote:
> Right now Hadoop seems to refer to the bundle of the MRv1 framework
> and HDFS though both can be used independently. Since MRv2 is its own
> top level project, where does leave HDFS?
>
> --Tom
>
> On Thu, Oct 18, 2012 at 5:01 PM, Ruslan Al-Fakikh <[EMAIL PROTECTED]> wrote:
>> Hi
>> From the beginning YARN was a part of the Hadoop project. Now it has
>> been decided to move it to a top-level apache project.
>> So, now there are two ways of running hadoop jobs:
>> 1) "Old" hadoop (the current stable thing)
>> 2) YARN + MR implementation (the future thing, still not stable AFAIK)
>>
>> Hope that helps
>>
>> On Fri, Oct 19, 2012 at 12:33 AM, Tom Brown <[EMAIL PROTECTED]> wrote:
>>> To all,
>>>
>>> I have a few questions regarding YARN (with respect to Hadoop):
>>>
>>> Are YARN and Hadoop separate, or is YARN the successor to Hadoop?
>>>
>>> What are the major conceptual differences between YARN and Hadoop?
>>>
>>> I have further questions, but they may or may not make sense depending
>>> on the answers to the above.
>>>
>>> Thanks in advance!
>>>
>>> Tom Brown

--
Harsh J

Thanks Harsh,

I have been playing with MapReduce v2 (YARN) in pseudo-dist mode and are trying to deploy it now on a cluster. This is what I came across on CD4 website:

Is this an up-to-date note? Shall I switch to MapReducev1 (MRv1) instead?

QTE
Note: Cloudera does not consider the current upstream MRv2 release stable yet, and it could potentially change in non-backwards-compatible ways. Cloudera recommends that you use MRv1 unless you have particular reasons for using MRv2, which should not be considered production-ready.
UNQTE

-----Original Message-----
From: Harsh J [mailto:[EMAIL PROTECTED]]
Sent: Friday, October 19, 2012 1:34 AM
To: [EMAIL PROTECTED]
Subject: Re: Differences between YARN and Hadoop

Andy,

"YARN" is NOT "MRv2". That seems to be a major confusion with people thanks to everyone mixing up the terms. See this small post for a clear difference:
http://www.cloudera.com/blog/2012/10/mr2-and-yarn-briefly-explained/

On Fri, Oct 19, 2012 at 3:30 AM, Kartashov, Andy <[EMAIL PROTECTED]> wrote:
> They are not comparable.
>
> YARN also known as MRv2 is the newer version of MapReduce also known as MRv1.
>
>
>
>
> -----Original Message-----
> From: Tom Brown [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, October 18, 2012 4:33 PM
> To: [EMAIL PROTECTED]
> Subject: Differences between YARN and Hadoop
>
> To all,
>
> I have a few questions regarding YARN (with respect to Hadoop):
>
> Are YARN and Hadoop separate, or is YARN the successor to Hadoop?
>
> What are the major conceptual differences between YARN and Hadoop?
>
> I have further questions, but they may or may not make sense depending on the answers to the above.
>
> Thanks in advance!
>
> Tom Brown
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use,
> copying or disclosure is prohibited. If you are not the intended
> recipient, please delete and contact the sender immediately. Please
> consider the environment before printing this e-mail. AVIS : le
> présent courriel et toute pièce jointe qui l'accompagne sont
> confidentiels, protégés par le droit d'auteur et peuvent être couverts
> par le secret professionnel. Toute utilisation, copie ou divulgation
> non autorisée est interdite. Si vous n'êtes pas le destinataire prévu
> de ce courriel, supprimez-le et contactez immédiatement l'expéditeur.
> Veuillez penser à l'environnement avant d'imprimer le présent courriel

--
Harsh J
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel