Hi there

Is there a way to disable retries when a mapper/reducer fails? I'm writing data in my mapper and I'd rather catch the failure, recover from a backup (fairly lightweight in this case, as the output tables aren't big) and restart.

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

Hi Marco

You can disable retries by setting
mapred.map.max.attempts and mapred.reduce.max.attempts  to 1.

Also if you need to disable speculative execution. You can disable it by setting
mapred.map.tasks.speculative.execution and mapred.reduce.tasks.speculative.execution to false.

With these two steps you can ensure that a task is attempted only once.

These properties to be set in mapred-site.xml or at job level.
Regards
Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: Marco Gallotta <[EMAIL PROTECTED]>
Date: Thu, 2 Aug 2012 16:52:00
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Disable retries

Hi there

Is there a way to disable retries when a mapper/reducer fails? I'm writing data in my mapper and I'd rather catch the failure, recover from a backup (fairly lightweight in this case, as the output tables aren't big) and restart.

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

Hi. Marco gallotta,
   Yes , there are some properties available but i had not tried . Let me
know if not works.
   If you using yarn their properties where you specify max retries
" yarn.resourcemanager.am.max-retries" and in reducer end there is a
properies and for node manager that u can specifed per node job.maxtaskfai
lures.per.tracker

If its hadoop 1.0 then "mapred.map.max.attempts" and
"mapred.reduce.max.attempts"

Syed Abdul kather
send from Samsung S3
On Aug 3, 2012 5:22 AM, "Marco Gallotta" <[EMAIL PROTECTED]> wrote:

> Hi there
>
> Is there a way to disable retries when a mapper/reducer fails? I'm writing
> data in my mapper and I'd rather catch the failure, recover from a backup
> (fairly lightweight in this case, as the output tables aren't big) and
> restart.
>
>
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> [EMAIL PROTECTED] | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>

Great, thanks!

--
Marco Gallotta | Mountain View, California
Software Engineer, Infrastructure | Loki Studios
fb.me/marco.gallotta | twitter.com/marcog
[EMAIL PROTECTED] | +1 (650) 417-3313

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Thursday 02 August 2012 at 5:02 PM, Bejoy KS wrote:

> Hi Marco
>
> You can disable retries by setting
> mapred.map.max.attempts and mapred.reduce.max.attempts to 1.
>
> Also if you need to disable speculative execution. You can disable it by setting
> mapred.map.tasks.speculative.execution and mapred.reduce.tasks.speculative.execution to false.
>
> With these two steps you can ensure that a task is attempted only once.
>
> These properties to be set in mapred-site.xml or at job level.
>
>
> Regards
> Bejoy KS
>
> Sent from handheld, please excuse typos.
>
> -----Original Message-----
> From: Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> Date: Thu, 2 Aug 2012 16:52:00
> To: <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> Reply-To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> Subject: Disable retries
>
> Hi there
>
> Is there a way to disable retries when a mapper/reducer fails? I'm writing data in my mapper and I'd rather catch the failure, recover from a backup (fairly lightweight in this case, as the output tables aren't big) and restart.
>
>
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

You may use the APIs directly:
http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/mapred/JobConf.html#setMaxMapAttempts(int)
and http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/mapred/JobConf.html#setMaxReduceAttempts(int)
to avoid config strings pain.

On Fri, Aug 3, 2012 at 5:59 AM, Marco Gallotta <[EMAIL PROTECTED]> wrote:
> Great, thanks!
>
> --
> Marco Gallotta | Mountain View, California
> Software Engineer, Infrastructure | Loki Studios
> fb.me/marco.gallotta | twitter.com/marcog
> [EMAIL PROTECTED] | +1 (650) 417-3313
>
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Thursday 02 August 2012 at 5:02 PM, Bejoy KS wrote:
>
>> Hi Marco
>>
>> You can disable retries by setting
>> mapred.map.max.attempts and mapred.reduce.max.attempts to 1.
>>
>> Also if you need to disable speculative execution. You can disable it by setting
>> mapred.map.tasks.speculative.execution and mapred.reduce.tasks.speculative.execution to false.
>>
>> With these two steps you can ensure that a task is attempted only once.
>>
>> These properties to be set in mapred-site.xml or at job level.
>>
>>
>> Regards
>> Bejoy KS
>>
>> Sent from handheld, please excuse typos.
>>
>> -----Original Message-----
>> From: Marco Gallotta <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
>> Date: Thu, 2 Aug 2012 16:52:00
>> To: <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
>> Reply-To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
>> Subject: Disable retries
>>
>> Hi there
>>
>> Is there a way to disable retries when a mapper/reducer fails? I'm writing data in my mapper and I'd rather catch the failure, recover from a backup (fairly lightweight in this case, as the output tables aren't big) and restart.
>>
>>
>>
>> --
>> Marco Gallotta | Mountain View, California
>> Software Engineer, Infrastructure | Loki Studios
>> fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog (http://twitter.com/marcog)
>> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) | +1 (650) 417-3313
>>
>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>

--
Harsh J