Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> MulitpleOutputs outputs just one line

Copy link to this message
MulitpleOutputs outputs just one line

I'm trying to utilize MulitpleOutputs ( hadoop 1.0.4 ) to produce multiple
files based on some policy. In the job i set:

MultipleOutputs.addNamedOutput( job, "rejected", TextOutputFormat.class,
Text.class, NullWritable.class );

And at the mapper:

private MultipleOutputs<Text, Writable> mos;

setup(): mos = new MultipleOutputs( context );

map():   if( somecond )
                     context.write( new Text( key ), NullWritable.get() );
                     logger.info( "Going to write to mos: " + key );
                     mos.write( new Text( key ), NullWritable.get(), "/tmp"

The problem I'm facing is that if multiple mappers running that code, I can
see at the logs that the mos.write() is being invoked, but only one line is
printed to the output file under /tmp. Is there some config I missed?

Harsh J 2013-01-24, 07:36
Barak Yaish 2013-01-24, 07:42