Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - OutputCommitter for rollbacks?


Copy link to this message
-
Re: OutputCommitter for rollbacks?
Amareshwari Sriramadasu 2009-08-10, 04:05
Hi Deepika,

You can use the fact that map progress and reduce progress 1.0 for
succeeded jobs and is <1.0 for failed or killed jobs.
Hope this helps.

Thanks
Amareshwari

Deepika Khera wrote:
> Hi,
>
>  
>
> I am trying to use the OutputCommitter.cleanupJob() to commit and
> rollback my job. The cleanup() method is called whether the job was
> successful, killed or failed.
>
>  
>
> I need to know in the cleanupJob(jobContext) method the status of job so
> far, especially if it failed or was killed.
>
>  
>
> The issue is that no matter whether the job failed or was
> killed/successful, the job status in the committer is "running" (which
> makes sense but not what I need). It seems that I can only know the job
> status when the entire job (including the cleanup task) has finished.
>
>  
>
> I see an open JIRA  which is related , but until we have a resolution to
> that is there any other  way to achieve this?
>
>  
>
> http://issues.apache.org/jira/browse/HADOOP-6005
>
>  
>
> So, what I need is make the decision in the cleanupJob() whether I
> should do  a commit or rollback(Killed/Failed vs Successful).
>
>  
>
> Would appreciate any help on this.
>
>  
>
> Thanks,
>
> Deepika
>
>
>