Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> how to set diffent VM parameters for mappers and reducers?


Copy link to this message
-
Re: how to set diffent VM parameters for mappers and reducers?
You can set mapred.child.java.opts in mapred-site.xml

BTW, combiner can been run both in map side and reduce side
On Tue, Oct 5, 2010 at 8:59 PM, Vitaliy Semochkin <[EMAIL PROTECTED]> wrote:
> Hello,
>
>
> I have mappers that do not need much ram but combiners and reducers need a lot.
> Is it possible to set different VM parameters for mappers and reducers?
>
>
>
> PS Often I face interesting problem, on same set of data I
> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
> but it happens not all the time.
> What could be cause of such behavior?
> My personal opinion is that I have
>
> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
> it should.
>
> Thanks in Advance,
> Vitaliy S
>

--
Best Regards

Jeff Zhang
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB