Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> how to set diffent VM parameters for mappers and reducers?


Copy link to this message
-
Re: how to set diffent VM parameters for mappers and reducers?
You can set mapred.child.java.opts in mapred-site.xml

BTW, combiner can been run both in map side and reduce side
On Tue, Oct 5, 2010 at 8:59 PM, Vitaliy Semochkin <[EMAIL PROTECTED]> wrote:
> Hello,
>
>
> I have mappers that do not need much ram but combiners and reducers need a lot.
> Is it possible to set different VM parameters for mappers and reducers?
>
>
>
> PS Often I face interesting problem, on same set of data I
> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
> but it happens not all the time.
> What could be cause of such behavior?
> My personal opinion is that I have
>
> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
> it should.
>
> Thanks in Advance,
> Vitaliy S
>

--
Best Regards

Jeff Zhang