Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Streaming Job map/reduce not working with scripts on 1.0.3


Copy link to this message
-
Re: Streaming Job map/reduce not working with scripts on 1.0.3
nevermind. the problem has been fixed.

The problem was the trailing {control-v}{control-m} character on the first
line of #!/bin/bash
(which i blame my teammate for writing the script in windows notepad !!)

On Fri, Jan 4, 2013 at 8:09 PM, Ben Kim <[EMAIL PROTECTED]> wrote:

> Hi !
>
> I'm using hadoop-1.0.3 to run streaming jobs with map/reduce shell scripts
> such as this
>
> bin/hadoop jar ./contrib/streaming/hadoop-streaming-1.0.3.jar -input
> /input -output /output/015 -mapper "streaming-map.sh" -reducer
> "streaming-reduce.sh" -file /home/hadoop/streaming/streaming-map.sh -file
> /home/hadoop/streaming-reduce.sh
>
> but the job fails and the task attemp log shows this,
>
> java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 9 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> ... 14 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 17 more
> Caused by: java.lang.RuntimeException: configuration exception
> at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:230)
> at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
> ... 22 more
> Caused by: java.io.IOException: Cannot run program "/tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201301041944_0001/attempt_201301041944_0001_m_000000_0/work/./streaming-map.sh": java.io.IOException: error=2, No such file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
> at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
> ... 23 more
> Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
> ... 24 more
>
>
> I tried to see what the problem is and found out that the missing file is
> a symbolic link and hadoop isn't able to create it, in fact the
> /tmp/hadoop-hadoop/...........00000_0/work directory doesn't exist at all.
>
>
> here's an exerpt from the task attempt syslog (full text attached):
>
> 2013-01-04 19:44:43,304 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201301041944_0001/jars/streaming-map.sh <- /tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201301041944_0001/attempt_201301041944_0001_m_000000_0/work/streaming-map.sh
*Benjamin Kim*
*benkimkimben at gmail*
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB