Hello,
I just noticed that the files that are created using MultipleOutputs remain in the temporary folder into attempt sub-folders when there is no normal output  (using context.write(...)).

Has anyone else noticed that?
Is there any way to change that and make the files appear in the output directory?

Thank you in advance!
Panagiotis.
     

Panayotis,

I've not seen this happen yet. I've regularly used MO to write my
files and both TextFileO/F and NullO/F have worked fine despite me not
writing a byte to their collectors. In fact, the test case for MO too
passes when I modify it to never emit to the default output sink.

Are you using the default OutputCommitter (FileOutputCommitter)?

2011/5/30 Panayotis Antonopoulos <[EMAIL PROTECTED]>:
> Hello,
> I just noticed that the files that are created using MultipleOutputs remain
> in the temporary folder into attempt sub-folders when there is no normal
> output  (using context.write(...)).
>
> Has anyone else noticed that?
> Is there any way to change that and make the files appear in the output
> directory?
>
> Thank you in advance!
> Panagiotis.
>

--
Harsh J

On 05/30/2011 11:02 AM, Panayotis Antonopoulos wrote:
> Hello,
> I just noticed that the files that are created using MultipleOutputs
> remain in the temporary folder into attempt sub-folders when there is
> no normal output  (using context.write(...)).
>
> Has anyone else noticed that?
> Is there any way to change that and make the files appear in the
> output directory?
>
> Thank you in advance!
> Panagiotis.
        |mapred.local.dir|

This lets the MapReduce servers know where to store intermediate files.
This may be a comma-separated list of directories to spread the load.
Make sure thereοΏ½s enough space here for all your intermediate files. We
share the same disks for MapReduce and HDFS.
        |mapred.system.dir|

This is a folder in the|defaultFS|where MapReduce stores some control
files. In our case that would be a directory in HDFS. If you
have|dfs.permissions|(which it is by default) enabled make sure that
this directory exists and is owned by mapred:hadoop.
        |mapred.temp.dir|

This is a folder to store temporary files in. It is hardly -- if at all
used. If I understand the description correctly this is supposed to be
in HDFS but IοΏ½m not entirely sure by reading the source code. So we set
this to a directory that exists on the local filesystem as well as in HDFS.

--
Marcos Luis Ortiz Valmaseda
  Software Engineer (Distributed Systems)
  http://uncubanitolinuxero.blogspot.com

Thanks for your replies!

I use TableOutputFormat to delete entries from a HBase Table and MO for HFileOutputFormat.
Until yesterday I used normal HFileOutputFormat output (not MO) and the files appeared in the output directory and not in the temporary folder.
I performed the deletes manually inside the reducer.

Then I decided to use MO for the HFiles and TableOutputFormat for the deletes.
Now the Hfiles remain in the temporary folder.

I think that TableOutputFormat  uses its own committer and I do not explicitly change the commiter.

Do you have anything to suggest?

> From: [EMAIL PROTECTED]
> Date: Tue, 31 May 2011 01:20:57 +0530
> Subject: Re: MultipleOutputs Files remain in temporary folder
> To: [EMAIL PROTECTED]
>
> Panayotis,
>
> I've not seen this happen yet. I've regularly used MO to write my
> files and both TextFileO/F and NullO/F have worked fine despite me not
> writing a byte to their collectors. In fact, the test case for MO too
> passes when I modify it to never emit to the default output sink.
>
> Are you using the default OutputCommitter (FileOutputCommitter)?
>
> 2011/5/30 Panayotis Antonopoulos <[EMAIL PROTECTED]>:
> > Hello,
> > I just noticed that the files that are created using MultipleOutputs remain
> > in the temporary folder into attempt sub-folders when there is no normal
> > output  (using context.write(...)).
> >
> > Has anyone else noticed that?
> > Is there any way to change that and make the files appear in the output
> > directory?
> >
> > Thank you in advance!
> > Panagiotis.
> >
>
>
>
> --
> Harsh J

     

I forgot to mention that there is no _SUCCESS folder in the output directory.

Thanks for your replies!

I use TableOutputFormat to delete entries from a HBase Table and MO for HFileOutputFormat.
Until yesterday I used normal HFileOutputFormat output (not MO) and the files appeared in the output directory and not in the temporary folder.
I performed the deletes manually inside the reducer.

Then I decided to use MO for the HFiles and TableOutputFormat for the deletes.
Now the Hfiles remain in the temporary folder.

I think that TableOutputFormat  uses its own committer and I do not explicitly change the commiter.

Do you have anything to suggest?

> From: [EMAIL PROTECTED]
> Date: Tue, 31 May 2011 01:20:57 +0530
> Subject: Re: MultipleOutputs Files remain in temporary folder
> To: [EMAIL PROTECTED]
>
> Panayotis,
>
> I've not seen this happen yet. I've regularly used MO to write my
> files and both TextFileO/F and NullO/F have worked fine despite me not
> writing a byte to their collectors. In fact, the test case for MO too
> passes when I modify it to never emit to the default output sink.
>
> Are you using the default OutputCommitter (FileOutputCommitter)?
>
> 2011/5/30 Panayotis Antonopoulos <[EMAIL PROTECTED]>:
> > Hello,
> > I just noticed that the files that are created using MultipleOutputs remain
> > in the temporary folder into attempt sub-folders when there is no normal
> > output  (using context.write(...)).
> >
> > Has anyone else noticed that?
> > Is there any way to change that and make the files appear in the output
> > directory?
> >
> > Thank you in advance!
> > Panagiotis.
> >
>
>
>
> --
> Harsh J
     

I found the solution to the problem!
As Harsh said I didn't use FileOutputCommiter because this Commiter is not used in TableOutputFormat.
I changed the TableOutputFormat class to use this commiter and now everything works great!

Thank you for you help!!!

From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: RE: MultipleOutputs Files remain in temporary folder
Date: Tue, 31 May 2011 05:37:27 +0300
I forgot to mention that there is no _SUCCESS folder in the output directory.

Thanks for your replies!

I use TableOutputFormat to delete entries from a HBase Table and MO for HFileOutputFormat.
Until yesterday I used normal HFileOutputFormat output (not MO) and the files appeared in the output directory and not in the temporary folder.
I performed the deletes manually inside the reducer.

Then I decided to use MO for the HFiles and TableOutputFormat for the deletes.
Now the Hfiles remain in the temporary folder.

I think that TableOutputFormat  uses its own committer and I do not explicitly change the commiter.

Do you have anything to suggest?

> From: [EMAIL PROTECTED]
> Date: Tue, 31 May 2011 01:20:57 +0530
> Subject: Re: MultipleOutputs Files remain in temporary folder
> To: [EMAIL PROTECTED]
>
> Panayotis,
>
> I've not seen this happen yet. I've regularly used MO to write my
> files and both TextFileO/F and NullO/F have worked fine despite me not
> writing a byte to their collectors. In fact, the test case for MO too
> passes when I modify it to never emit to the default output sink.
>
> Are you using the default OutputCommitter (FileOutputCommitter)?
>
> 2011/5/30 Panayotis Antonopoulos <[EMAIL PROTECTED]>:
> > Hello,
> > I just noticed that the files that are created using MultipleOutputs remain
> > in the temporary folder into attempt sub-folders when there is no normal
> > output  (using context.write(...)).
> >
> > Has anyone else noticed that?
> > Is there any way to change that and make the files appear in the output
> > directory?
> >
> > Thank you in advance!
> > Panagiotis.
> >
>
>
>
> --
> Harsh J