Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> MultipleOutputFormat


Copy link to this message
-
Re: MultipleOutputFormat
You should be able to create partitions on the fly.

Check the last example in the javadocs:

http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.html

 ...

  mos.write(key, new Text("value"), generateFileName(key, new Text("value")));

Hope this helps.

Alejandro

On Wed, Mar 30, 2011 at 12:02 PM, Dmitriy Lyubimov <[EMAIL PROTECTED]>wrote:

> yes.. but in my old code the file names are created on the fly (it
> basically creates partitions based on a time field). I dont think
> MultipleOutputs is not suitable to create partitions on the fly.
>
> On Tue, Mar 29, 2011 at 8:56 PM, Alejandro Abdelnur <[EMAIL PROTECTED]>
> wrote:
> > Dmitriy,
> > Have you check the MultipleOutputs instead? It provides similar
> > functionality.
> > Alejandro
> >
> > On Wed, Mar 30, 2011 at 11:39 AM, Dmitriy Lyubimov <[EMAIL PROTECTED]>
> > wrote:
> >>
> >> Hi,
> >> I can't seem to be able to find either jira or implementation of
> >> MultipleOutputFormat in new api in either 0.21 or 0.22 branches.
> >> Are there any plans to port that to new api as well?
> >>
> >> thanks in advance.
> >> -Dmitriy
> >
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB