Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - MultipleOutputFormat


Copy link to this message
-
Re: MultipleOutputFormat
Alejandro Abdelnur 2011-03-30, 04:09
You should be able to create partitions on the fly.

Check the last example in the javadocs:

http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.html

 ...

  mos.write(key, new Text("value"), generateFileName(key, new Text("value")));

Hope this helps.

Alejandro

On Wed, Mar 30, 2011 at 12:02 PM, Dmitriy Lyubimov <[EMAIL PROTECTED]>wrote:

> yes.. but in my old code the file names are created on the fly (it
> basically creates partitions based on a time field). I dont think
> MultipleOutputs is not suitable to create partitions on the fly.
>
> On Tue, Mar 29, 2011 at 8:56 PM, Alejandro Abdelnur <[EMAIL PROTECTED]>
> wrote:
> > Dmitriy,
> > Have you check the MultipleOutputs instead? It provides similar
> > functionality.
> > Alejandro
> >
> > On Wed, Mar 30, 2011 at 11:39 AM, Dmitriy Lyubimov <[EMAIL PROTECTED]>
> > wrote:
> >>
> >> Hi,
> >> I can't seem to be able to find either jira or implementation of
> >> MultipleOutputFormat in new api in either 0.21 or 0.22 branches.
> >> Are there any plans to port that to new api as well?
> >>
> >> thanks in advance.
> >> -Dmitriy
> >
> >
>