Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - JobClient: Error reading task output - after instituting a DNS server


Copy link to this message
-
JobClient: Error reading task output - after instituting a DNS server
David Parks 2013-05-14, 06:19
So we just configured a local DNS server for hostname resolution and stopped
using a hosts file and now jobs fail on us. But I can't figure out why.

 

You can see the error below, but if I run curl to any of those URLs they
come back "Failed to retrieve stdout log", which doesn't look much like a
DNS issue.

 

I can ping and do nslookup from any host to any other host. This is a CDH4
cluster and the host inspector is happy as could be; also Cloudera Manager
indicates all is well.

 

When I open the task tracker website I see the first task attempt show up on
the site there for maybe 10 seconds or so before it fails.

 

Any idea what I need to look at here?

 

Job:

===
13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process :
131

13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process :
1

13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003

13/05/14 05:13:43 INFO mapred.JobClient:  map 0% reduce 0%

13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_m_000353_0, Status : FAILED

java.lang.Throwable: Child Error

        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)

Caused by: java.io.IOException: Task process exit with nonzero status of 1.

        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)

 

13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stdout

13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stderr

13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_r_000521_0, Status : FAILED

java.lang.Throwable: Child Error

        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)

Caused by: java.io.IOException: Task process exit with nonzero status of 1.

        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)

 

13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stdout

13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stderr

 

 

curl of above URL:

===================
davidparks21@hadoop-meta1:~$ curl
'http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_2013
05131758_0003_m_000353_0&filter=stdout'

<html>

<head>

<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>

<title>Error 410 Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</title>

</head>

<body><h2>HTTP ERROR 410</h2>

<p>Problem accessing /tasklog. Reason:

<pre>    Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
Jetty://</small></i><br/>                                

<br/>

<br/>

<br/>

<br/>

+
David Parks 2013-05-14, 06:29
+
Shahab Yunus 2013-05-14, 11:55
+
David Parks 2013-05-15, 07:28