Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: Skipping entire task


Copy link to this message
-
Re: Skipping entire task
Håvard Wahl Kongsgård 2013-01-06, 15:29
Thanks, I was unaware of mapred.max.map.failures.percent

-Håvard

On Sun, Jan 6, 2013 at 3:46 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> You can use the mapred.max.map.failures.percent and
> mapred.max.reduce.failures.percent features to control the percentage
> of allowed failures of tasks in a single job (despite which the job is
> marked successful).
>
> On Sun, Jan 6, 2013 at 8:04 PM, Håvard Wahl Kongsgård
> <[EMAIL PROTECTED]> wrote:
>>> Are tasks being executed multiple times due to failures? Sorry, it was not
>>> very clear from your question.
>>
>> yes, and I simply want to skip them if they fail more than x
>> times(after all this is big data :) ).
>>
>> -Håvard
>>
>> On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
>> <[EMAIL PROTECTED]> wrote:
>>> Hi,
>>>
>>> Are tasks being executed multiple times due to failures? Sorry, it was not
>>> very clear from your question.
>>>
>>> Thanks
>>> hemanth
>>>
>>>
>>> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <[EMAIL PROTECTED]> wrote:
>>>>
>>>> Thinking here... if you submitted the task programmatically you should be
>>>> able to capture the failure of the task and gracefully move past it to
>>>> your
>>>> next tasks.
>>>>
>>>> To say it in a long-winded way:  Let's say you submit a job to Hadoop, a
>>>> java jar, and your main class implements Tool. That code has the
>>>> responsibility to submit a series of jobs to hadoop, something like this:
>>>>
>>>> try{
>>>>   Job myJob = new MyJob(getConf());
>>>>   myJob.submitAndWait();
>>>> }catch(Exception uhhohh){
>>>>   //Deal with the issue and move on
>>>> }
>>>> Job myNextJob = new MyNextJob(getConf());
>>>> myNextJob.submit();
>>>>
>>>> Just pseudo code there to demonstrate my thought.
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Håvard Wahl Kongsgård [mailto:[EMAIL PROTECTED]]
>>>> Sent: Saturday, January 05, 2013 4:54 PM
>>>> To: user
>>>> Subject: Skipping entire task
>>>>
>>>> Hi, hadoop can skip bad records
>>>>
>>>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
>>>> ode.
>>>> But it is also possible to skip entire tasks?
>>>>
>>>> -Håvard
>>>>
>>>> --
>>>> Håvard Wahl Kongsgård
>>>> Faculty of Medicine &
>>>> Department of Mathematical Sciences
>>>> NTNU
>>>>
>>>> http://havard.security-review.net/
>>>>
>>>
>>
>>
>>
>> --
>> Håvard Wahl Kongsgård
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.security-review.net/
>
>
>
> --
> Harsh J

--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU

http://havard.security-review.net/