Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: Why my tests shows Yarn is worse than MRv1 for terasort?


+
sam liu 2013-06-07, 03:15
+
Sandy Ryza 2013-06-07, 06:53
+
Harsh J 2013-06-08, 15:09
+
sam liu 2013-06-09, 08:51
Copy link to this message
-
Re: Why my tests shows Yarn is worse than MRv1 for terasort?
Hi Sam,

> - How to know the container number? Why you say it will be 22 containers due to a 22 GB memory?

The MR2's default configuration requests 1 GB resource each for Map
and Reduce containers. It requests 1.5 GB for the AM container that
runs the job, additionally. This is tunable using the properties
Sandy's mentioned in his post.

> - My machine has 32 GB memory, how many memory is proper to be assigned to containers?

This is a general question. You may use the same process you took to
decide optimal number of slots in MR1 to decide this here. Every
container is a new JVM, and you're limited by the CPUs you have there
(if not the memory). Either increase memory requests from jobs, to
lower # of concurrent containers at a given time (runtime change), or
lower NM's published memory resources to control the same (config
change).

> - In mapred-site.xml, if I set 'mapreduce.framework.name' to be 'yarn', will other parameters for mapred-site.xml still work in yarn framework? Like 'mapreduce.task.io.sort.mb' and 'mapreduce.map.sort.spill.percent'

Yes, all of these properties will still work. Old properties specific
to JobTracker or TaskTracker (usually found as a keyword in the config
name) will not apply anymore.

On Sun, Jun 9, 2013 at 2:21 PM, sam liu <[EMAIL PROTECTED]> wrote:
> Hi Harsh,
>
> According to above suggestions, I removed the duplication of setting, and
> reduce the value of 'yarn.nodemanager.resource.cpu-cores',
> 'yarn.nodemanager.vcores-pcores-ratio' and
> 'yarn.nodemanager.resource.memory-mb' to 16, 8 and 12000. Ant then, the
> efficiency improved about 18%.  I have questions:
>
> - How to know the container number? Why you say it will be 22 containers due
> to a 22 GB memory?
> - My machine has 32 GB memory, how many memory is proper to be assigned to
> containers?
> - In mapred-site.xml, if I set 'mapreduce.framework.name' to be 'yarn', will
> other parameters for mapred-site.xml still work in yarn framework? Like
> 'mapreduce.task.io.sort.mb' and 'mapreduce.map.sort.spill.percent'
>
> Thanks!
>
>
>
> 2013/6/8 Harsh J <[EMAIL PROTECTED]>
>>
>> Hey Sam,
>>
>> Did you get a chance to retry with Sandy's suggestions? The config
>> appears to be asking NMs to use roughly 22 total containers (as
>> opposed to 12 total tasks in MR1 config) due to a 22 GB memory
>> resource. This could impact much, given the CPU is still the same for
>> both test runs.
>>
>> On Fri, Jun 7, 2013 at 12:23 PM, Sandy Ryza <[EMAIL PROTECTED]>
>> wrote:
>> > Hey Sam,
>> >
>> > Thanks for sharing your results.  I'm definitely curious about what's
>> > causing the difference.
>> >
>> > A couple observations:
>> > It looks like you've got yarn.nodemanager.resource.memory-mb in there
>> > twice
>> > with two different values.
>> >
>> > Your max JVM memory of 1000 MB is (dangerously?) close to the default
>> > mapreduce.map/reduce.memory.mb of 1024 MB. Are any of your tasks getting
>> > killed for running over resource limits?
>> >
>> > -Sandy
>> >
>> >
>> > On Thu, Jun 6, 2013 at 10:21 PM, sam liu <[EMAIL PROTECTED]> wrote:
>> >>
>> >> The terasort execution log shows that reduce spent about 5.5 mins from
>> >> 33%
>> >> to 35% as below.
>> >> 13/06/10 08:02:22 INFO mapreduce.Job:  map 100% reduce 31%
>> >> 13/06/10 08:02:25 INFO mapreduce.Job:  map 100% reduce 32%
>> >> 13/06/10 08:02:46 INFO mapreduce.Job:  map 100% reduce 33%
>> >> 13/06/10 08:08:16 INFO mapreduce.Job:  map 100% reduce 35%
>> >> 13/06/10 08:08:19 INFO mapreduce.Job:  map 100% reduce 40%
>> >> 13/06/10 08:08:22 INFO mapreduce.Job:  map 100% reduce 43%
>> >>
>> >> Any way, below are my configurations for your reference. Thanks!
>> >> (A) core-site.xml
>> >> only define 'fs.default.name' and 'hadoop.tmp.dir'
>> >>
>> >> (B) hdfs-site.xml
>> >>   <property>
>> >>     <name>dfs.replication</name>
>> >>     <value>1</value>
>> >>   </property>
>> >>
>> >>   <property>
>> >>     <name>dfs.name.dir</name>
>> >>     <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/dfs_name_dir</value>

Harsh J
+
sam liu 2013-06-18, 08:58
+
Michel Segel 2013-06-18, 10:11
+
Sandy Ryza 2013-10-22, 23:45
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB