Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> FileOutputCommiter behavior doubts


+
Subroto 2012-05-23, 09:23
+
Harsh J 2012-05-23, 11:18
Copy link to this message
-
Re: FileOutputCommiter behavior doubts
Thanks Harsh….
Filed MAPREDUCE-4280 for the same….
Cheers,
Subroto Sanyal

On May 23, 2012, at 1:18 PM, Harsh J wrote:

> This is related: https://issues.apache.org/jira/browse/MAPREDUCE-2493
>
> But the real issue is LocalJobRunner does:
>
> OutputCommitter outputCommitter = job.getOutputCommitter();
>
> Which doesn't work with New API cause the OutputCommitter now comes
> from within the OutputFormat class. So hence you get the default
> FileOutputCommitter returned from configuration, and hence your issue.
>
> Can you file a JIRA for this? We'll have to fix LocalJobRunner to do
> things differently for new API usage.
>
> On Wed, May 23, 2012 at 2:53 PM, Subroto <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>> I have an implementation SampleFileOutputCommiter which
>> extends org.apache.hadoop.mapred.FileOutputCommitter . The implementation
>> has specific code to be executed during cleanupJob() execution.
>> When the framework(LocalJobRunner) makes a call to commitJob(), the
>> framework never takes care of calling the cleanupJob() instead it
>> calls org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitJob()
>> which internally calls its own cleanupJob().
>> Though the method cleanupJob is deprecated but, still I feel the framework
>> should take care of executing it as it is being executed
>> from org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.
>> Currently the framework is not letting the Jobs written with MRV1 to run
>> properly.
>>
>> Any thoughts….. ??
>>
>> Cheers,
>> Subroto Sanyal
>
>
>
> --
> Harsh J

+
Harsh J 2012-05-23, 11:43
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB