Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: pipes on hadoop 2.2.0 crashes


Copy link to this message
-
Re: pipes on hadoop 2.2.0 crashes
I tested that example as well and I'm getting the same exception with this
in stderr:

Hadoop Pipes Exception: failed to open hdfs://compute-0-7-2:54310/in/file
at
/home/scaino/hadoop-2.2.0-maven/hadoop-tools/hadoop-pipes/src/main/native/examples/impl/wordcount-nopipe.cc:82
in WordCountReader

I found out that wordcount-nopipe is picky for its input format, here is
the full explanation:
http://lucene.472066.n3.nabble.com/pipe-application-error-td650185.html
On 17 December 2013 17:36, Mauro Del Rio <[EMAIL PROTECTED]> wrote:

> Ok, I had some problems with configuration and host resolution and I fixed
> them. I was able to run successfully the simple wordcount example, but I
> failed running wordcount-nopipe.cc.
> This is the stack trace:
>
> Error: java.io.IOException: pipe child exception
> at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:226)
> at
> org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:102)
>  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
> at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:250)
>  at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
>  at
> org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:125)
>
>
> Any suggestions?
>
>
> 2013/12/17 Silvina Caíno Lores <[EMAIL PROTECTED]>
>
>> I'm having similar problems with pipes, mostly because of issues with the
>> native shared libraries that leave the job stuck either at 0%-0% or before
>> launch (because the resource manager gets stuck as well and crashes).
>>
>> I found that out by looking at the stderr logs by the way.
>>
>> Let us know if you progress, good luck!
>>
>> Best,
>> Silvina
>>
>>
>> On 16 December 2013 19:32, Vinod Kumar Vavilapalli <
>> [EMAIL PROTECTED]> wrote:
>>
>>> You should navigate to the ResourceManager UI following the link and see
>>> what is happening on the ResourceManager as well as the application-master.
>>> Check if any nodes are active first. Then look at ResourceManager and
>>> NodeManager logs.
>>>
>>>  +Vinod
>>>
>>> On Dec 16, 2013, at 10:29 AM, Mauro Del Rio <[EMAIL PROTECTED]> wrote:
>>>
>>> I installed hadoop 2.2.0 on a small cluster, just two nodes. I run a
>>> simple wordcount in c++ with pipes, this time there was no exception, but
>>> the job didn't finish. This is the output on the shell where I launched
>>> pipes:
>>>
>>> mauro@mauro-VirtualBox:~/hadoop-2.2.0$ bin/mapred pipes -program wc
>>> -input test.sh -output out
>>> 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at /
>>> 0.0.0.0:9052
>>> 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at /
>>> 0.0.0.0:9052
>>> 13/12/16 18:51:41 WARN mapreduce.JobSubmitter: No job jar file set.
>>>  User classes may not be found. See Job or Job#setJar(String).
>>> 13/12/16 18:51:41 INFO mapred.FileInputFormat: Total input paths to
>>> process : 1
>>> 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: number of splits:2
>>> 13/12/16 18:51:41 INFO Configuration.deprecation: user.name is
>>> deprecated. Instead, use mapreduce.job.user.name
>>> 13/12/16 18:51:41 INFO Configuration.deprecation:
>>> mapred.cache.files.filesizes is deprecated. Instead, use
>>> mapreduce.job.cache.files.filesizes
>>> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files is
>>> deprecated. Instead, use mapreduce.job.cache.files
>>> 13/12/16 18:51:41 INFO Configuration.deprecation:
>>> mapred.pipes.user.inputformat is deprecated. Instead, use
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB