Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: Increase the number of mappers in PM mode


Copy link to this message
-
Re: Increase the number of mappers in PM mode
Harsh J 2013-03-17, 00:00
In MR2, to have more mappers executed per NM, your memory request for each
map should be set such that the NM's configured memory allowance can fit in
multiple requests. For example, if my NM memory is set to 16 GB assuming
just 1 NM in cluster, and I submit a job with mapreduce.map.memory.mb and
yarn.app.mapreduce.am.resource.mb both set to 1 GB, then the NM can execute
15 maps in parallel consuming upto 1 GB memory each (while using the
remaining 1 GB for the AM to coordinate those executions).
On Sat, Mar 16, 2013 at 10:16 AM, yypvsxf19870706 <[EMAIL PROTECTED]
> wrote:

> hi:
>    i think i have got it . Thank you.
>
> 发自我的 iPhone
>
> 在 2013-3-15,18:32,Zheyi RONG <[EMAIL PROTECTED]> 写道:
>
> Indeed you cannot explicitly set the number of mappers, but still you can
> gain some control over it, by setting mapred.max.split.size, or
> mapred.min.split.size.
>
> For example, if you have a file of 10GB (10737418240 B), you would like 10
> mappers, then each mapper has to deal with 1GB data.
> According to "splitsize = max(minimumSize, min(maximumSize, blockSize))",
> you can set mapred.min.split.size=1073741824 (1GB), i.e.
> $hadoop jar -Dmapred.min.split.size=1073741824 yourjar yourargs
>
> It is well explained in thread:
> http://stackoverflow.com/questions/9678180/change-file-split-size-in-hadoop
> .
>
> Regards,
> Zheyi.
>
> On Fri, Mar 15, 2013 at 8:49 AM, YouPeng Yang <[EMAIL PROTECTED]>wrote:
>
>> s
>
>
>
>
--
Harsh J
+
Zheyi RONG 2013-03-14, 14:30
+
YouPeng Yang 2013-03-15, 07:49
+
Zheyi RONG 2013-03-15, 10:32