Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: Container size configuration


Copy link to this message
-
Re: Container size configuration
You mean different memory sizes for different reducers?  If you're willing
to mess with the MR Application Master code, you should be to do it there.
 Check out RMContainerAllocator.java and RMContainerRequestor.java.

-Sandy
On Thu, Jun 13, 2013 at 9:56 AM, Yuzhang Han <[EMAIL PROTECTED]>wrote:

>  Thank you Sandy.
>
> I am trying to do to some experiments with MapReduce on YARN, where some
> reducers gain larger processing tasks than the other reducers. I want to
> see that if I allocate more memory to the larger-task reducers, how the
> performance would be improved.
>
> Anyway, if I want to manually configure memory size for a particular
> reducer, how can I do it? In ApplicationMaster? Or this would not work at
> all?
>
> Thank you.
>
>
>
>
>
> On 6/13/2013 12:47 PM, Sandy Ryza wrote:
>
> Hi Yuzhang,
>
>  Moving this question to the Hadoop user list.
>
>  Are you using MapReduce or writing your own YARN application?  In
> MapReduce, all maps must request the same amount of memory and all reduces
> must request the same amount of memory.  It would be trivial to do this in
> your own YARN application.
>
>  -Sandy
>
>
> On Thu, Jun 13, 2013 at 8:41 AM, Yuzhang Han <[EMAIL PROTECTED]>wrote:
>
>> Hi,
>>
>> I am wondering if I can allocate different size of containers to the
>> tasks in a job. For example: Job = <Task1, Task2, Task3>, Task1 = Task2 >> 1024MB, Task3 = 2048MB. How can I achieve this? Many thanks.
>>
>> Yuzhang
>>
>
>
>