Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Skipping entire task


Copy link to this message
-
Re: Skipping entire task
Thanks, I was unaware of mapred.max.map.failures.percent

-Håvard

On Sun, Jan 6, 2013 at 3:46 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> You can use the mapred.max.map.failures.percent and
> mapred.max.reduce.failures.percent features to control the percentage
> of allowed failures of tasks in a single job (despite which the job is
> marked successful).
>
> On Sun, Jan 6, 2013 at 8:04 PM, Håvard Wahl Kongsgård
> <[EMAIL PROTECTED]> wrote:
>>> Are tasks being executed multiple times due to failures? Sorry, it was not
>>> very clear from your question.
>>
>> yes, and I simply want to skip them if they fail more than x
>> times(after all this is big data :) ).
>>
>> -Håvard
>>
>> On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
>> <[EMAIL PROTECTED]> wrote:
>>> Hi,
>>>
>>> Are tasks being executed multiple times due to failures? Sorry, it was not
>>> very clear from your question.
>>>
>>> Thanks
>>> hemanth
>>>
>>>
>>> On Sat, Jan 5, 2013 at 7:44 PM, David Parks <[EMAIL PROTECTED]> wrote:
>>>>
>>>> Thinking here... if you submitted the task programmatically you should be
>>>> able to capture the failure of the task and gracefully move past it to
>>>> your
>>>> next tasks.
>>>>
>>>> To say it in a long-winded way:  Let's say you submit a job to Hadoop, a
>>>> java jar, and your main class implements Tool. That code has the
>>>> responsibility to submit a series of jobs to hadoop, something like this:
>>>>
>>>> try{
>>>>   Job myJob = new MyJob(getConf());
>>>>   myJob.submitAndWait();
>>>> }catch(Exception uhhohh){
>>>>   //Deal with the issue and move on
>>>> }
>>>> Job myNextJob = new MyNextJob(getConf());
>>>> myNextJob.submit();
>>>>
>>>> Just pseudo code there to demonstrate my thought.
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Håvard Wahl Kongsgård [mailto:[EMAIL PROTECTED]]
>>>> Sent: Saturday, January 05, 2013 4:54 PM
>>>> To: user
>>>> Subject: Skipping entire task
>>>>
>>>> Hi, hadoop can skip bad records
>>>>
>>>> http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-c
>>>> ode.
>>>> But it is also possible to skip entire tasks?
>>>>
>>>> -Håvard
>>>>
>>>> --
>>>> Håvard Wahl Kongsgård
>>>> Faculty of Medicine &
>>>> Department of Mathematical Sciences
>>>> NTNU
>>>>
>>>> http://havard.security-review.net/
>>>>
>>>
>>
>>
>>
>> --
>> Håvard Wahl Kongsgård
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.security-review.net/
>
>
>
> --
> Harsh J

--
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU

http://havard.security-review.net/
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB