Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> question about using java in streaming mode


Copy link to this message
-
Re: question about using java in streaming mode
El 6/5/2011 4:01 PM, Siddhartha Jonnalagadda escribi�:
> Hi Marcos,
>
> I thought that streaming would make it easier because I was getting
> different errors with extending mapper and reducer in java.
>
> I tried: hadoop jar contrib/streaming/hadoop-streaming-0.20.203.0.jar
> -file dummy.jar -mapper "java -cp dummy.jar WCMapper" -reducer "java
> -cp dummy.jar WCReducer" -input gutenberg/* -output gutenberg-output
>
> The error log in the map task:
> *_stderr logs_*
> Exception in thread "main" java.lang.NoClassDefFoundError: WCMapper
> Caused by: java.lang.ClassNotFoundException: WCMapper
>    
Which is the definition of your ClassPath? Because, this error is caused
where the system can not find
the definition of a class.
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
>
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>
>
> Could not find the main class: WCMapper.  Program will exit.
> java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
> at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
>
>
> at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
> at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:121)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
>
>
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
>
>
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
> java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
>
>
> at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
> at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
> at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
>
>
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
>
>
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>
>
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>    
> ------------------------------------------------------------------------
>
>
>
> Sincerely,
> Siddhartha Jonnalagadda,
> sjonnalagadda.wordpress.com <http://sjonnalagadda.wordpress.com>
>
>
> Confidentiality Notice:
>
> This e-mail message, including any attachments, is for the sole use of
> the intended recipient(s) and may contain confidential and privileged
> information. Any unauthorized review, use, disclosure or distribution
> is prohibited. If you are not the intended recipient, please contact
> the sender by reply e-mail and destroy all copies of the original message.
>
>
>
>
>
>
> On Sun, Jun 5, 2011 at 10:59 AM, Marcos Ortiz Valmaseda
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
>
>     Why are using Java in streming mode instead use the native
>     Mapper/Reducer code?
>     Can you show to us the JobTracker's logs?
>
>     Regards
>     ----- Mensaje original -----
>     De: "Siddhartha Jonnalagadda" <[EMAIL PROTECTED]
Marcos Lu�s Ort�z Valmaseda
  Software Engineer (UCI)
  http://marcosluis2186.posterous.com
  http://twitter.com/marcosluis2186