Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> question about using java in streaming mode


Copy link to this message
-
Re: question about using java in streaming mode
El 6/5/2011 4:01 PM, Siddhartha Jonnalagadda escribi�:
> Hi Marcos,
>
> I thought that streaming would make it easier because I was getting
> different errors with extending mapper and reducer in java.
>
> I tried: hadoop jar contrib/streaming/hadoop-streaming-0.20.203.0.jar
> -file dummy.jar -mapper "java -cp dummy.jar WCMapper" -reducer "java
> -cp dummy.jar WCReducer" -input gutenberg/* -output gutenberg-output
>
> The error log in the map task:
> *_stderr logs_*
> Exception in thread "main" java.lang.NoClassDefFoundError: WCMapper
> Caused by: java.lang.ClassNotFoundException: WCMapper
>    
Which is the definition of your ClassPath? Because, this error is caused
where the system can not find
the definition of a class.
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
>
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>
>
> Could not find the main class: WCMapper.  Program will exit.
> java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
> at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
>
>
> at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
> at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:121)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
>
>
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
>
>
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
> java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
>
>
> at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
> at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
> at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
>
>
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
>
>
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>
>
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>    
> ------------------------------------------------------------------------
>
>
>
> Sincerely,
> Siddhartha Jonnalagadda,
> sjonnalagadda.wordpress.com <http://sjonnalagadda.wordpress.com>
>
>
> Confidentiality Notice:
>
> This e-mail message, including any attachments, is for the sole use of
> the intended recipient(s) and may contain confidential and privileged
> information. Any unauthorized review, use, disclosure or distribution
> is prohibited. If you are not the intended recipient, please contact
> the sender by reply e-mail and destroy all copies of the original message.
>
>
>
>
>
>
> On Sun, Jun 5, 2011 at 10:59 AM, Marcos Ortiz Valmaseda
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
>
>     Why are using Java in streming mode instead use the native
>     Mapper/Reducer code?
>     Can you show to us the JobTracker's logs?
>
>     Regards
>     ----- Mensaje original -----
>     De: "Siddhartha Jonnalagadda" <[EMAIL PROTECTED]
Marcos Lu�s Ort�z Valmaseda
  Software Engineer (UCI)
  http://marcosluis2186.posterous.com
  http://twitter.com/marcosluis2186
  
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB