Hello,
I have mappers that do not need much ram but combiners and reducers need a lot.
Is it possible to set different VM parameters for mappers and reducers?

PS Often I face interesting problem, on same set of data I
recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
but it happens not all the time.
What could be cause of such behavior?
My personal opinion is that I have

mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
it should.

Thanks in Advance,
Vitaliy S

Hi,

You don't say which version of Hadoop you are using.
Going from memory, I believe in the CDH3 release from Cloudera, there are some specific OPTs you can set in hadoop-env.sh.

HTH

-Mike
> Date: Tue, 5 Oct 2010 16:59:35 +0400
> Subject: how to set diffent VM parameters for mappers and reducers?
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
> Hello,
>
>
> I have mappers that do not need much ram but combiners and reducers need a lot.
> Is it possible to set different VM parameters for mappers and reducers?
>
>
>
> PS Often I face interesting problem, on same set of data I
> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
> but it happens not all the time.
> What could be cause of such behavior?
> My personal opinion is that I have
>
> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
> it should.
>
> Thanks in Advance,
> Vitaliy S
     

The following 2 properties should work:

mapred.map.child.java.opts
mapred.reduce.child.java.opts

Alejandro
On Tue, Oct 5, 2010 at 9:02 PM, Michael Segel <[EMAIL PROTECTED]> wrote:
>
> Hi,
>
> You don't say which version of Hadoop you are using.
> Going from memory, I believe in the CDH3 release from Cloudera, there are some specific OPTs you can set in hadoop-env.sh.
>
> HTH
>
> -Mike
>
>
>> Date: Tue, 5 Oct 2010 16:59:35 +0400
>> Subject: how to set diffent VM parameters for mappers and reducers?
>> From: [EMAIL PROTECTED]
>> To: [EMAIL PROTECTED]
>>
>> Hello,
>>
>>
>> I have mappers that do not need much ram but combiners and reducers need a lot.
>> Is it possible to set different VM parameters for mappers and reducers?
>>
>>
>>
>> PS Often I face interesting problem, on same set of data I
>> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
>> but it happens not all the time.
>> What could be cause of such behavior?
>> My personal opinion is that I have
>>
>> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
>> it should.
>>
>> Thanks in Advance,
>> Vitaliy S
>

You can set mapred.child.java.opts in mapred-site.xml

BTW, combiner can been run both in map side and reduce side
On Tue, Oct 5, 2010 at 8:59 PM, Vitaliy Semochkin <[EMAIL PROTECTED]> wrote:
> Hello,
>
>
> I have mappers that do not need much ram but combiners and reducers need a lot.
> Is it possible to set different VM parameters for mappers and reducers?
>
>
>
> PS Often I face interesting problem, on same set of data I
> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
> but it happens not all the time.
> What could be cause of such behavior?
> My personal opinion is that I have
>
> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
> it should.
>
> Thanks in Advance,
> Vitaliy S
>

--
Best Regards

Jeff Zhang

I'm using apache hadoop-0.20.2 - the recent version i found in maven
central repo.

Regards,
Vitaliy S

On Tue, Oct 5, 2010 at 5:02 PM, Michael Segel <[EMAIL PROTECTED]> wrote:
>
> Hi,
>
> You don't say which version of Hadoop you are using.
> Going from memory, I believe in the CDH3 release from Cloudera, there are some specific OPTs you can set in hadoop-env.sh.
>
> HTH
>
> -Mike
>
>
>> Date: Tue, 5 Oct 2010 16:59:35 +0400
>> Subject: how to set diffent VM parameters for mappers and reducers?
>> From: [EMAIL PROTECTED]
>> To: [EMAIL PROTECTED]
>>
>> Hello,
>>
>>
>> I have mappers that do not need much ram but combiners and reducers need a lot.
>> Is it possible to set different VM parameters for mappers and reducers?
>>
>>
>>
>> PS Often I face interesting problem, on same set of data I
>> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
>> but it happens not all the time.
>> What could be cause of such behavior?
>> My personal opinion is that I have
>>
>> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
>> it should.
>>
>> Thanks in Advance,
>> Vitaliy S
>

Hi,

I tried using  mapred.map.child.java.opts and mapred.reduce.child.java.opts
but looks like  hadoop-0.20.2  ingnores it.

On which version have you seen it working?

Regards,
Vitaliy S

On Tue, Oct 5, 2010 at 5:14 PM, Alejandro Abdelnur <[EMAIL PROTECTED]> wrote:
> The following 2 properties should work:
>
> mapred.map.child.java.opts
> mapred.reduce.child.java.opts
>
> Alejandro
>
>
> On Tue, Oct 5, 2010 at 9:02 PM, Michael Segel <[EMAIL PROTECTED]> wrote:
>>
>> Hi,
>>
>> You don't say which version of Hadoop you are using.
>> Going from memory, I believe in the CDH3 release from Cloudera, there are some specific OPTs you can set in hadoop-env.sh.
>>
>> HTH
>>
>> -Mike
>>
>>
>>> Date: Tue, 5 Oct 2010 16:59:35 +0400
>>> Subject: how to set diffent VM parameters for mappers and reducers?
>>> From: [EMAIL PROTECTED]
>>> To: [EMAIL PROTECTED]
>>>
>>> Hello,
>>>
>>>
>>> I have mappers that do not need much ram but combiners and reducers need a lot.
>>> Is it possible to set different VM parameters for mappers and reducers?
>>>
>>>
>>>
>>> PS Often I face interesting problem, on same set of data I
>>> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
>>> but it happens not all the time.
>>> What could be cause of such behavior?
>>> My personal opinion is that I have
>>>
>>> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
>>> it should.
>>>
>>> Thanks in Advance,
>>> Vitaliy S
>>
>

Never used those myself, always used the global one, but I knew they are
there.

Which Hadoop API are you using, the old one or the new one?

Alejandro

On Fri, Oct 8, 2010 at 3:50 AM, Vitaliy Semochkin <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I tried using  mapred.map.child.java.opts and mapred.reduce.child.java.opts
> but looks like  hadoop-0.20.2  ingnores it.
>
> On which version have you seen it working?
>
> Regards,
> Vitaliy S
>
> On Tue, Oct 5, 2010 at 5:14 PM, Alejandro Abdelnur <[EMAIL PROTECTED]>
> wrote:
> > The following 2 properties should work:
> >
> > mapred.map.child.java.opts
> > mapred.reduce.child.java.opts
> >
> > Alejandro
> >
> >
> > On Tue, Oct 5, 2010 at 9:02 PM, Michael Segel <[EMAIL PROTECTED]>
> wrote:
> >>
> >> Hi,
> >>
> >> You don't say which version of Hadoop you are using.
> >> Going from memory, I believe in the CDH3 release from Cloudera, there
> are some specific OPTs you can set in hadoop-env.sh.
> >>
> >> HTH
> >>
> >> -Mike
> >>
> >>
> >>> Date: Tue, 5 Oct 2010 16:59:35 +0400
> >>> Subject: how to set diffent VM parameters for mappers and reducers?
> >>> From: [EMAIL PROTECTED]
> >>> To: [EMAIL PROTECTED]
> >>>
> >>> Hello,
> >>>
> >>>
> >>> I have mappers that do not need much ram but combiners and reducers
> need a lot.
> >>> Is it possible to set different VM parameters for mappers and reducers?
> >>>
> >>>
> >>>
> >>> PS Often I face interesting problem, on same set of data I
> >>> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
> >>> but it happens not all the time.
> >>> What could be cause of such behavior?
> >>> My personal opinion is that I have
> >>>
> >>> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
> >>> it should.
> >>>
> >>> Thanks in Advance,
> >>> Vitaliy S
> >>
> >
>

>You can set mapred.child.java.opts in mapred-site.xml
it affects both Mappers and Reducers, while I need to specify jvm
options for mappers and reducers separately.

Regards,
Vitaliy S

On Tue, Oct 5, 2010 at 5:14 PM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> You can set mapred.child.java.opts in mapred-site.xml
>
> BTW, combiner can been run both in map side and reduce side
>
>
>
>
> On Tue, Oct 5, 2010 at 8:59 PM, Vitaliy Semochkin <[EMAIL PROTECTED]> wrote:
>> Hello,
>>
>>
>> I have mappers that do not need much ram but combiners and reducers need a lot.
>> Is it possible to set different VM parameters for mappers and reducers?
>>
>>
>>
>> PS Often I face interesting problem, on same set of data I
>> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
>> but it happens not all the time.
>> What could be cause of such behavior?
>> My personal opinion is that I have
>>
>> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
>> it should.
>>
>> Thanks in Advance,
>> Vitaliy S
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

I use new API and  hadoop-0.20.2

On which distribution have you seen it working?

Thanks in Advance,
Vitaliy S

On Fri, Oct 8, 2010 at 3:08 AM, Alejandro Abdelnur <[EMAIL PROTECTED]> wrote:
> Never used those myself, always used the global one, but I knew they are
> there.
>
> Which Hadoop API are you using, the old one or the new one?
>
> Alejandro
>
> On Fri, Oct 8, 2010 at 3:50 AM, Vitaliy Semochkin <[EMAIL PROTECTED]>wrote:
>
>> Hi,
>>
>> I tried using  mapred.map.child.java.opts and mapred.reduce.child.java.opts
>> but looks like  hadoop-0.20.2  ingnores it.
>>
>> On which version have you seen it working?
>>
>> Regards,
>> Vitaliy S
>>
>> On Tue, Oct 5, 2010 at 5:14 PM, Alejandro Abdelnur <[EMAIL PROTECTED]>
>> wrote:
>> > The following 2 properties should work:
>> >
>> > mapred.map.child.java.opts
>> > mapred.reduce.child.java.opts
>> >
>> > Alejandro
>> >
>> >
>> > On Tue, Oct 5, 2010 at 9:02 PM, Michael Segel <[EMAIL PROTECTED]>
>> wrote:
>> >>
>> >> Hi,
>> >>
>> >> You don't say which version of Hadoop you are using.
>> >> Going from memory, I believe in the CDH3 release from Cloudera, there
>> are some specific OPTs you can set in hadoop-env.sh.
>> >>
>> >> HTH
>> >>
>> >> -Mike
>> >>
>> >>
>> >>> Date: Tue, 5 Oct 2010 16:59:35 +0400
>> >>> Subject: how to set diffent VM parameters for mappers and reducers?
>> >>> From: [EMAIL PROTECTED]
>> >>> To: [EMAIL PROTECTED]
>> >>>
>> >>> Hello,
>> >>>
>> >>>
>> >>> I have mappers that do not need much ram but combiners and reducers
>> need a lot.
>> >>> Is it possible to set different VM parameters for mappers and reducers?
>> >>>
>> >>>
>> >>>
>> >>> PS Often I face interesting problem, on same set of data I
>> >>> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
>> >>> but it happens not all the time.
>> >>> What could be cause of such behavior?
>> >>> My personal opinion is that I have
>> >>>
>> >>> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
>> >>> it should.
>> >>>
>> >>> Thanks in Advance,
>> >>> Vitaliy S
>> >>
>> >
>>
>

I too need a solution for this as I am facing same problem.

Thanks in anticipation.
On Thu, Oct 7, 2010 at 3:50 PM, Vitaliy Semochkin <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I tried using  mapred.map.child.java.opts and mapred.reduce.child.java.opts
> but looks like  hadoop-0.20.2  ingnores it.
>
> On which version have you seen it working?
>
> Regards,
> Vitaliy S
>
> On Tue, Oct 5, 2010 at 5:14 PM, Alejandro Abdelnur <[EMAIL PROTECTED]> wrote:
>> The following 2 properties should work:
>>
>> mapred.map.child.java.opts
>> mapred.reduce.child.java.opts
>>
>> Alejandro
>>
>>
>> On Tue, Oct 5, 2010 at 9:02 PM, Michael Segel <[EMAIL PROTECTED]> wrote:
>>>
>>> Hi,
>>>
>>> You don't say which version of Hadoop you are using.
>>> Going from memory, I believe in the CDH3 release from Cloudera, there are some specific OPTs you can set in hadoop-env.sh.
>>>
>>> HTH
>>>
>>> -Mike
>>>
>>>
>>>> Date: Tue, 5 Oct 2010 16:59:35 +0400
>>>> Subject: how to set diffent VM parameters for mappers and reducers?
>>>> From: [EMAIL PROTECTED]
>>>> To: [EMAIL PROTECTED]
>>>>
>>>> Hello,
>>>>
>>>>
>>>> I have mappers that do not need much ram but combiners and reducers need a lot.
>>>> Is it possible to set different VM parameters for mappers and reducers?
>>>>
>>>>
>>>>
>>>> PS Often I face interesting problem, on same set of data I
>>>> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
>>>> but it happens not all the time.
>>>> What could be cause of such behavior?
>>>> My personal opinion is that I have
>>>>
>>>> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
>>>> it should.
>>>>
>>>> Thanks in Advance,
>>>> Vitaliy S
>>>
>>
>

I guess we should create  a feature request )

On Mon, Oct 11, 2010 at 7:34 AM, Medha Atre <[EMAIL PROTECTED]> wrote:
> I too need a solution for this as I am facing same problem.
>
> Thanks in anticipation.
>
>
> On Thu, Oct 7, 2010 at 3:50 PM, Vitaliy Semochkin <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>> I tried using  mapred.map.child.java.opts and mapred.reduce.child.java.opts
>> but looks like  hadoop-0.20.2  ingnores it.
>>
>> On which version have you seen it working?
>>
>> Regards,
>> Vitaliy S
>>
>> On Tue, Oct 5, 2010 at 5:14 PM, Alejandro Abdelnur <[EMAIL PROTECTED]> wrote:
>>> The following 2 properties should work:
>>>
>>> mapred.map.child.java.opts
>>> mapred.reduce.child.java.opts
>>>
>>> Alejandro
>>>
>>>
>>> On Tue, Oct 5, 2010 at 9:02 PM, Michael Segel <[EMAIL PROTECTED]> wrote:
>>>>
>>>> Hi,
>>>>
>>>> You don't say which version of Hadoop you are using.
>>>> Going from memory, I believe in the CDH3 release from Cloudera, there are some specific OPTs you can set in hadoop-env.sh.
>>>>
>>>> HTH
>>>>
>>>> -Mike
>>>>
>>>>
>>>>> Date: Tue, 5 Oct 2010 16:59:35 +0400
>>>>> Subject: how to set diffent VM parameters for mappers and reducers?
>>>>> From: [EMAIL PROTECTED]
>>>>> To: [EMAIL PROTECTED]
>>>>>
>>>>> Hello,
>>>>>
>>>>>
>>>>> I have mappers that do not need much ram but combiners and reducers need a lot.
>>>>> Is it possible to set different VM parameters for mappers and reducers?
>>>>>
>>>>>
>>>>>
>>>>> PS Often I face interesting problem, on same set of data I
>>>>> recieve I have java.lang.OutOfMemoryError: Java heap space in combiner
>>>>> but it happens not all the time.
>>>>> What could be cause of such behavior?
>>>>> My personal opinion is that I have
>>>>>
>>>>> mapred.job.reuse.jvm.num.tasks=-1 and jvm GC doesn't always start when
>>>>> it should.
>>>>>
>>>>> Thanks in Advance,
>>>>> Vitaliy S
>>>>
>>>
>>
>

On 11/10/10 11:13, Vitaliy Semochkin wrote:
> I guess we should create  a feature request )

no, If it doesn't work it is a bug. Therefore you should

-get the latest released version of the code, that is 0.21
-search for the string there, check they are both actually being used,
see what happens to them.
-file a bug report
-write some MR job which would verify that the settings are passed down
-attach it as a test case.

Hadoop is a community project, which means if you want a feature fixed,
on your schedule, you get to fix it.

It's going to be hard to test this in the normal LocalMRCluster tests,
as they all run in-JVM, but there is work going on in the -dev lists on
better testing; they could the tests

-steve

Steve,

I'm working with hadoop - 0.20.2
was trhis feature available in hadoop 0.20.2 or it is a new feature of 0.21?

Regards,
Vitaliy S

2010/10/11 Steve Loughran <[EMAIL PROTECTED]>:
> On 11/10/10 11:13, Vitaliy Semochkin wrote:
>> I guess we should create  a feature request )
>
> no, If it doesn't work it is a bug. Therefore you should
>
> -get the latest released version of the code, that is 0.21
> -search for the string there, check they are both actually being used,
> see what happens to them.
> -file a bug report
> -write some MR job which would verify that the settings are passed down
> -attach it as a test case.
>
> Hadoop is a community project, which means if you want a feature fixed,
> on your schedule, you get to fix it.
>
> It's going to be hard to test this in the normal LocalMRCluster tests,
> as they all run in-JVM, but there is work going on in the -dev lists on
> better testing; they could the tests
>
> -steve
>

On 12/10/10 08:54, Vitaliy Semochkin wrote:
> Steve,
>
> I'm working with hadoop - 0.20.2
> was trhis feature available in hadoop 0.20.2 or it is a new feature of 0.21?
>
> Regards,
> Vitaliy S

I have no idea. You'll need to check the docs and then the source.

One other thing you'll need to do is make sure that you are setting the
options in the right place. If they are TaskTracker-specific, there's no
point setting them in the job configuration, as the TT doesn't read them
from there.

>
>
>
> 2010/10/11 Steve Loughran<[EMAIL PROTECTED]>:
>> On 11/10/10 11:13, Vitaliy Semochkin wrote:
>>> I guess we should create  a feature request )
>>
>> no, If it doesn't work it is a bug. Therefore you should
>>
>> -get the latest released version of the code, that is 0.21
>> -search for the string there, check they are both actually being used,
>> see what happens to them.
>> -file a bug report
>> -write some MR job which would verify that the settings are passed down
>> -attach it as a test case.
>>
>> Hadoop is a community project, which means if you want a feature fixed,
>> on your schedule, you get to fix it.
>>
>> It's going to be hard to test this in the normal LocalMRCluster tests,
>> as they all run in-JVM, but there is work going on in the -dev lists on
>> better testing; they could the tests
>>
>> -steve
>>

For 0.20.x, you can specify java opts for your tasks with:

mapred.child.java.opts for all tasks
mapred.map.child.java.opts (mapreduce.map.java.opts for 0.21+) for maps
mapred.reduce.child.java.opts (mapreduce.reduce.java.opts for 0.21+)
for reduces.

__Luke

On Tue, Oct 12, 2010 at 12:54 AM, Vitaliy Semochkin
<[EMAIL PROTECTED]> wrote:
> Steve,
>
> I'm working with hadoop - 0.20.2
> was trhis feature available in hadoop 0.20.2 or it is a new feature of 0.21?
>
> Regards,
> Vitaliy S
>
>
>
> 2010/10/11 Steve Loughran <[EMAIL PROTECTED]>:
>> On 11/10/10 11:13, Vitaliy Semochkin wrote:
>>> I guess we should create  a feature request )
>>
>> no, If it doesn't work it is a bug. Therefore you should
>>
>> -get the latest released version of the code, that is 0.21
>> -search for the string there, check they are both actually being used,
>> see what happens to them.
>> -file a bug report
>> -write some MR job which would verify that the settings are passed down
>> -attach it as a test case.
>>
>> Hadoop is a community project, which means if you want a feature fixed,
>> on your schedule, you get to fix it.
>>
>> It's going to be hard to test this in the normal LocalMRCluster tests,
>> as they all run in-JVM, but there is work going on in the -dev lists on
>> better testing; they could the tests
>>
>> -steve
>>
>