Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Streaming with user-defined Java driver


Copy link to this message
-
Streaming with user-defined Java driver
I would like to use streaming for the mapper and reducer but in such a way that I still get to do the usual Hadoop setup in Java, the usual Java arrangement, like this:

public class Coadd extends Configured implements Tool {
public int run(String[] args) throws Exception {
JobConf conf = new JobConf(getConf(), getClass());

...A BUNCH OF STUFF...

JobClient.runJob(conf);
}

public static void main(String[] args) throws Exception {
ToolRunner.run(new Coadd(), args);
}
}

How do I do this?  If I specify the streaming jar as the entry point then I don't see how to maintain the ability to define my own driver.

Thanks.

________________________________________________________________________________
Keith Wiley               [EMAIL PROTECTED]               www.keithwiley.com

"I do not feel obliged to believe that the same God who has endowed us with
sense, reason, and intellect has intended us to forgo their use."
  -- Galileo Galilei
________________________________________________________________________________