-Re: only one map or reduce job per time on one node
DSuiter RDX 2013-11-06, 14:43
I suspect that the reason no-one is responding with good answers is that
fundamentally, it seems like what you are trying to do runs against the
reason Hadoop is designed the way it is. A parallel process framework is
defeated if you force it to not work concurrently...
Maybe you should look into Oozie, and set a workflow that runs your map
task as a map-only job, and then do the calculation that the reducer would
do in a different program?
There is a Hadoop parameter that allows for task JVM reuse that might solve
your problem also.
Or, add a combiner to your mr job if there is not one already, and then try
dropping the maximum heap sizes of the respective slots?
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com
On Wed, Nov 6, 2013 at 5:09 AM, John <[EMAIL PROTECTED]> wrote:
> Does no one have an idea if it is possible? In my example the heapsize of
> the tasktracker is 2gb. The max map heap size is 1gb and the mac heap size
> of the reduce process 1,5gb. If there are running 2 maps or just the reduce
> job everything is fine. But if there are running 2 maps + 1 reduce process
> I got a "Error: java.lang.OutOfMemoryError: Java heap space at" message.
> kind regards
> 2013/11/5 John <[EMAIL PROTECTED]>
>> Because my node swaps the memory if the 2 map slots + 1 reduce is
>> occupied with my job. Sure I can minimize the max memory for the map/reduce
>> process. I tried this already, but I got a out of memory exception if set
>> the max heap size for the map/reduce process to low for my mr job.
>> kind regards
>> 2013/11/5 Vinod Kumar Vavilapalli <[EMAIL PROTECTED]>
>>> Why do you want to do this?
>>> On Nov 5, 2013, at 9:17 AM, John wrote:
>>> Is it possible to force the jobtracker executing only 2 map jobs or 1
>>> reduce job per time?
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.