Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: pipes on hadoop 2.2.0 crashes


Copy link to this message
-
Re: pipes on hadoop 2.2.0 crashes
I tested that example as well and I'm getting the same exception with this
in stderr:

Hadoop Pipes Exception: failed to open hdfs://compute-0-7-2:54310/in/file
at
/home/scaino/hadoop-2.2.0-maven/hadoop-tools/hadoop-pipes/src/main/native/examples/impl/wordcount-nopipe.cc:82
in WordCountReader

I found out that wordcount-nopipe is picky for its input format, here is
the full explanation:
http://lucene.472066.n3.nabble.com/pipe-application-error-td650185.html
On 17 December 2013 17:36, Mauro Del Rio <[EMAIL PROTECTED]> wrote:

> Ok, I had some problems with configuration and host resolution and I fixed
> them. I was able to run successfully the simple wordcount example, but I
> failed running wordcount-nopipe.cc.
> This is the stack trace:
>
> Error: java.io.IOException: pipe child exception
> at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:226)
> at
> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:102)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
> at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
>  at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
>  at
> org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:125)
>
>
> Any suggestions?
>
>
> 2013/12/17 Silvina Caíno Lores <[EMAIL PROTECTED]>
>
>> I'm having similar problems with pipes, mostly because of issues with the
>> native shared libraries that leave the job stuck either at 0%-0% or before
>> launch (because the resource manager gets stuck as well and crashes).
>>
>> I found that out by looking at the stderr logs by the way.
>>
>> Let us know if you progress, good luck!
>>
>> Best,
>> Silvina
>>
>>
>> On 16 December 2013 19:32, Vinod Kumar Vavilapalli <
>> [EMAIL PROTECTED]> wrote:
>>
>>> You should navigate to the ResourceManager UI following the link and see
>>> what is happening on the ResourceManager as well as the application-master.
>>> Check if any nodes are active first. Then look at ResourceManager and
>>> NodeManager logs.
>>>
>>>  +Vinod
>>>
>>> On Dec 16, 2013, at 10:29 AM, Mauro Del Rio <[EMAIL PROTECTED]> wrote:
>>>
>>> I installed hadoop 2.2.0 on a small cluster, just two nodes. I run a
>>> simple wordcount in c++ with pipes, this time there was no exception, but
>>> the job didn't finish. This is the output on the shell where I launched
>>> pipes:
>>>
>>> mauro@mauro-VirtualBox:~/hadoop-2.2.0$ bin/mapred pipes -program wc
>>> -input test.sh -output out
>>> 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at /
>>> 0.0.0.0:9052
>>> 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at /
>>> 0.0.0.0:9052
>>> 13/12/16 18:51:41 WARN mapreduce.JobSubmitter: No job jar file set.
>>>  User classes may not be found. See Job or Job#setJar(String).
>>> 13/12/16 18:51:41 INFO mapred.FileInputFormat: Total input paths to
>>> process : 1
>>> 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: number of splits:2
>>> 13/12/16 18:51:41 INFO Configuration.deprecation: user.name is
>>> deprecated. Instead, use mapreduce.job.user.name
>>> 13/12/16 18:51:41 INFO Configuration.deprecation:
>>> mapred.cache.files.filesizes is deprecated. Instead, use
>>> mapreduce.job.cache.files.filesizes
>>> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files is
>>> deprecated. Instead, use mapreduce.job.cache.files
>>> 13/12/16 18:51:41 INFO Configuration.deprecation:
>>> mapred.pipes.user.inputformat is deprecated. Instead, use