Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> OutputCommitter for rollbacks?


Copy link to this message
-
Re: OutputCommitter for rollbacks?
Deepika Khera wrote:
> Thanks Amareshwari for your response.
>
> It seems like a good idea to use the map progress & reduce progress. My
> only concern is that in the web interface(jobdetails.jsp) , we see some
> of our jobs show 100% map & 100% reduce, while the reduce still seems to
> be running(Not sure but maybe it's just a UI thing).
>
> But I guess if the job has reached its cleanup, we should be able to
> trust these numbers(map progress & reduce progress) and make the call on
> commit & rollback?
>
>  
Yes. You can assume that when cleanup is running and reduce progress is
100%, job is successful.
> Thanks again,
> Deepika
>
>
> -----Original Message-----
> From: Amareshwari Sriramadasu [mailto:[EMAIL PROTECTED]]
> Sent: Sunday, August 09, 2009 9:05 PM
> To: [EMAIL PROTECTED]
> Subject: Re: OutputCommitter for rollbacks?
>
> Hi Deepika,
>
> You can use the fact that map progress and reduce progress 1.0 for
> succeeded jobs and is <1.0 for failed or killed jobs.
> Hope this helps.
>
> Thanks
> Amareshwari
>
> Deepika Khera wrote:
>  
>> Hi,
>>
>>  
>>
>> I am trying to use the OutputCommitter.cleanupJob() to commit and
>> rollback my job. The cleanup() method is called whether the job was
>> successful, killed or failed.
>>
>>  
>>
>> I need to know in the cleanupJob(jobContext) method the status of job
>>    
> so
>  
>> far, especially if it failed or was killed.
>>
>>  
>>
>> The issue is that no matter whether the job failed or was
>> killed/successful, the job status in the committer is "running" (which
>> makes sense but not what I need). It seems that I can only know the
>>    
> job
>  
>> status when the entire job (including the cleanup task) has finished.
>>
>>  
>>
>> I see an open JIRA  which is related , but until we have a resolution
>>    
> to
>  
>> that is there any other  way to achieve this?
>>
>>  
>>
>> http://issues.apache.org/jira/browse/HADOOP-6005
>>
>>  
>>
>> So, what I need is make the decision in the cleanupJob() whether I
>> should do  a commit or rollback(Killed/Failed vs Successful).
>>
>>  
>>
>> Would appreciate any help on this.
>>
>>  
>>
>> Thanks,
>>
>> Deepika
>>
>>
>>  
>>    
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB