I'm not sure if Pig disables it or not, but I remember I had issues when that file was to be created by the MR jobs due to the fact of Oozie or Pig removing temporary directories or something like that. I remember seeing an exception about failure to create the SUCCESS file, so I started using the following property in my workflow pig action to disable that:
From: Rahul Bhattacharjee <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; [EMAIL PROTECTED]
Sent: Monday, May 6, 2013 3:48 AM
Subject: Re: no _SUCCESS file in MR output directory.
I wanted to confirm whether oozie disables the _SUCCESS file creation when
it triggers a MR job.
I am triggering a MR job (actually a bunch of 'em) from oozie and the
workflow completes successfully , however I do not see any kind of _SUCCESS
file in the output directory.
When I set the file output committer's configuration
) to true.It generates the success file. Wanted to confirm if oozie does
the disabling of success file creation.
On Mon, May 6, 2013 at 12:34 PM, Rahul Bhattacharjee <
[EMAIL PROTECTED]> wrote:
> Oozie is being used for triggering the MR job. Looks like oozie disables
> the success file creation using the configuration that you have mentioned
> for FileOutputCommitter.
> I have enabled it by setting this property in conf.
> On Mon, May 6, 2013 at 9:38 AM, Rahul Bhattacharjee <
> [EMAIL PROTECTED]> wrote:
>> Thanks Harsh for the pointers. I will find out more on this.
>> On Sun, May 5, 2013 at 11:26 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>> I can think of a few, most obvious ones:
>>> 1. Job didn't succeed and/or the file was deleted (*shields self*)
>>> 2. Job overrode the default FileOutputCommitter with something that
>>> doesn't do success marking.
>>> 3. Job specifically asked to not create such files, via config
>>> mapreduce.fileoutputcommitter.marksuccessfuljobs or so, set to false.
>>> On Sun, May 5, 2013 at 9:54 PM, Rahul Bhattacharjee
>>> <[EMAIL PROTECTED]> wrote:
>>> > Hi,
>>> > A few days back , I was going through a MR job's output , but there
>>> > any _SUCCESS file in the output directory.
>>> > I was wondering what all reasons for this (no _SUCCESS file)?
>>> > Thanks,
>>> > Rahul
>>> Harsh J