Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - only one map or reduce job per time on one node


Copy link to this message
-
Re: only one map or reduce job per time on one node
DSuiter RDX 2013-11-06, 14:43
I suspect that the reason no-one is responding with good answers is that
fundamentally, it seems like what you are trying to do runs against the
reason Hadoop is designed the way it is. A parallel process framework is
defeated if you force it to not work concurrently...

Maybe you should look into Oozie, and set a workflow that runs your map
task as a map-only job, and then do the calculation that the reducer would
do in a different program?

There is a Hadoop parameter that allows for task JVM reuse that might solve
your problem also.

Or, add a combiner to your mr job if there is not one already, and then try
dropping the maximum heap sizes of the respective slots?

*Devin Suiter*
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com
On Wed, Nov 6, 2013 at 5:09 AM, John <[EMAIL PROTECTED]> wrote:

> Does no one have an idea if it is possible? In my example the heapsize of
> the tasktracker is 2gb. The max map heap size is 1gb and the mac heap size
> of the reduce process 1,5gb. If there are running 2 maps or just the reduce
> job everything is fine. But if there are running 2 maps + 1 reduce process
> I got a "Error: java.lang.OutOfMemoryError: Java heap space at" message.
>
> kind regards
>
>
> 2013/11/5 John <[EMAIL PROTECTED]>
>
>> Because my node swaps the memory if the 2 map slots + 1 reduce is
>> occupied with my job. Sure I can minimize the max memory for the map/reduce
>> process. I tried this already, but I got a out of memory exception if set
>> the max heap size for the map/reduce process to low for my mr job.
>>
>> kind regards
>>
>>
>> 2013/11/5 Vinod Kumar Vavilapalli <[EMAIL PROTECTED]>
>>
>>> Why do you want to do this?
>>>
>>> +Vinod
>>>
>>> On Nov 5, 2013, at 9:17 AM, John wrote:
>>>
>>> Is it possible to force the jobtracker executing only 2 map jobs or 1
>>> reduce job per time?
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>