Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Streaming with user-defined Java driver


Copy link to this message
-
Streaming with user-defined Java driver
I would like to use streaming for the mapper and reducer but in such a way that I still get to do the usual Hadoop setup in Java, the usual Java arrangement, like this:

public class Coadd extends Configured implements Tool {
public int run(String[] args) throws Exception {
JobConf conf = new JobConf(getConf(), getClass());

...A BUNCH OF STUFF...

JobClient.runJob(conf);
}

public static void main(String[] args) throws Exception {
ToolRunner.run(new Coadd(), args);
}
}

How do I do this?  If I specify the streaming jar as the entry point then I don't see how to maintain the ability to define my own driver.

Thanks.

________________________________________________________________________________
Keith Wiley               [EMAIL PROTECTED]               www.keithwiley.com

"I do not feel obliged to believe that the same God who has endowed us with
sense, reason, and intellect has intended us to forgo their use."
  -- Galileo Galilei
________________________________________________________________________________
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB