Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> RE: Doubts: Deployment and Configuration of YARN cluster


+
Nirmal Kumar 2014-01-15, 14:52
+
Nirmal Kumar 2014-01-16, 07:03
Copy link to this message
-
Re: Doubts: Deployment and Configuration of YARN cluster

On Jan 16, 2014, at 9:14 PM, Nirmal Kumar <[EMAIL PROTECTED]> wrote:

> Hi Arun,
>  
> Thanks a lot for the clarification.
>  
> I understand it like in *yarn-site.xml* first I can set the max. and min. container size GLOBALLY for all the nodes in the cluster through:
> ·         {yarn.scheduler.minimum-allocation-mb}
> ·         {yarn.scheduler.maximum-allocation-mb}
>  
> Then at each of the node I can set the NM memory using:
> ·         {yarn.nodemanager.resource.memory-mb}
>  
>

Exactly! :)

> My 2nd doubt is whether we can run Mappers\Reducers tasks with varying memory options at each of the slave nodes.
> That is, can we change the following properties in *mapred-site.xml* at each of the slave nodes? This is because depending on the machine’s power we can adjust the memory options for the M\R tasks.
> ·         {mapreduce.map.memory.mb}
> ·         {mapreduce.map.java.opts}
> ·         {mapreduce.reduce.memory.mb}
> ·         {mapreduce.reduce.java.opts}

You can change these for every single job, so each job can have different requirements.

$ bin/hadoop jar hadoop-examples.jar word count -Dmapreduce.map.memory.mb=1024 …
$ bin/hadoop jar hadoop-examples.jar word count -Dmapreduce.map.memory.mb=2048 …

hth,
Arun

>  
> Thanks,
> -Nirmal
>  
> From: Arun C Murthy [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, January 16, 2014 7:43 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Doubts: Deployment and Configuration of YARN cluster
>  
> No, you can set resources available in each node to be different…
>  
> For e.g. Node A: 10G, Node B: 12G.
>  
> Now, if min. container size is 1G, the RM will allocate 10 containers to Node A and 12 containers to Node B.
>  
> hth,
> Arun
>  
> On Jan 15, 2014, at 11:03 PM, Nirmal Kumar <[EMAIL PROTECTED]> wrote:
>
>
> Hi German,
>  
> I went through the links for memory configuration settings/best-practices.
> It considers the cluster to be homogenous i.e. same RAM size in all the nodes.
>  
> Also on the Yarn whitepaper(Section 3.2 Page 6) I see:
> This resource model serves current applications well
> in homogeneous environments, but we expect it to
> evolve over time as the ecosystem matures and new requirements
> emerge.
>  
> Does that mean in YARN in order to configure processing capacity like Container Size, No. of Containers, No. of Mappers\Reducers the cluster has to be homogenous?
> How about if I have a *heterogeneous cluster* with varying RAM, disks , cores?
>  
> Thanks,
> -Nirmal
>  
> From: Nirmal Kumar
> Sent: Wednesday, January 15, 2014 8:22 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Doubts: Deployment and Configuration of YARN cluster
>  
> Thanks a lot German.
>  
> Will go through the links and see if that answers my questions\doubts.
>  
> -Nirmal
>  
> From: German Florez-Larrahondo [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, January 15, 2014 7:20 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Doubts: Deployment and Configuration of YARN cluster
>  
> Nirmal
>  
> -A good summary regarding memory configuration settings/best-practices can be found here. Note that in YARN, the way you configure resource limits dictates number of containers in the nodes and in the cluster:
> http://dev.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html
>  
> -A good intro to YARN configuration is this:
> http://www.thecloudavenue.com/2012/01/getting-started-with-nextgen-mapreduce_11.html
>  
> Regards
> .g
>  
>  
>  
> From: Nirmal Kumar [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, January 15, 2014 7:22 AM
> To: [EMAIL PROTECTED]
> Subject: Doubts: Deployment and Configuration of YARN cluster
>  
> All,
>  
> I am new to YARN and have certain doubts regarding the deployment and configuration of YARN on a cluster.
>  
> As per my understanding to deploy Hadoop 2.x using YARN on a cluster we need to distribute the below files to all the slave nodes in the cluster:
> ·         conf/core-site.xml

Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
+
Nirmal Kumar 2014-01-17, 16:20
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB