Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> OutputCommitter for rollbacks?


Copy link to this message
-
Re: OutputCommitter for rollbacks?
Hi Deepika,

You can use the fact that map progress and reduce progress 1.0 for
succeeded jobs and is <1.0 for failed or killed jobs.
Hope this helps.

Thanks
Amareshwari

Deepika Khera wrote:
> Hi,
>
>  
>
> I am trying to use the OutputCommitter.cleanupJob() to commit and
> rollback my job. The cleanup() method is called whether the job was
> successful, killed or failed.
>
>  
>
> I need to know in the cleanupJob(jobContext) method the status of job so
> far, especially if it failed or was killed.
>
>  
>
> The issue is that no matter whether the job failed or was
> killed/successful, the job status in the committer is "running" (which
> makes sense but not what I need). It seems that I can only know the job
> status when the entire job (including the cleanup task) has finished.
>
>  
>
> I see an open JIRA  which is related , but until we have a resolution to
> that is there any other  way to achieve this?
>
>  
>
> http://issues.apache.org/jira/browse/HADOOP-6005
>
>  
>
> So, what I need is make the decision in the cleanupJob() whether I
> should do  a commit or rollback(Killed/Failed vs Successful).
>
>  
>
> Would appreciate any help on this.
>
>  
>
> Thanks,
>
> Deepika
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB