I'm trying to get some hands-on on the Map Reduce programming.
I downloaded the code examples from Hadoop-The definitive guide, 3rd edition and build it using Maven: mvn package -DskipTests -Dhadoop.distro=apache-2
Next I imported the maven projects into Eclipse. Using Eclipse now I can develop my own Map Reduce jobs. Bu how do I test\run the job locally using the "Local Job Runner"?
The book excerpt says: Now we can run this application against some local files. Hadoop comes with a local job runner, a cut-down version of the MapReduce execution engine for running Map- Reduce jobs in a single JVM. It’s designed for testing, and is very convenient for use in an IDE, since you can run it in a debugger to step through the code in your mapper and reducer.
Do I also need to install Hadoop locally on Windows for that?
You don't need to have the actual Hadoop daemons running on windows machince. Just install Cygwin and ensure that you have all the required Hadoop jars in the class path of your program. You can test/debug directly from the IDE itself just by saying "Run As" -> "Java Application" on the driver class. This will run the program in the Local Job Runner mode. You can use this to verify the basic logic of your MR code. We you start using advanced features of MR, the behavior/output on Local Job Runner may be different from when running the program on a real distributed cluster.
Regards, Rakesh On Thu, Mar 6, 2014 at 2:36 PM, Radhe Radhe <[EMAIL PROTECTED]>wrote:
I just ran the Driver class from Eclipse and it worked fine.ðŸ˜Š
Can I use local job runner to test Sequence File and Map File as well ?ðŸ˜Œ
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext