Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - JobClient: Error reading task output - after instituting a DNS server


Copy link to this message
-
RE: JobClient: Error reading task output - after instituting a DNS server
David Parks 2013-05-15, 07:28
So simple I was hoping to avoid admitting to it. ;-)

 

I had set the tasks java options at -Xmx1.5g, that needed to be -Xmx1500m,
the telltale output of a mistake like that is rather tricky to find, I had
to dig into the task tracker UI/logs, it doesn't show up on the job
tracker's normal logs.

 

The timing perfectly coincided with a DNS change, and Googles first hit, on
the error that I *could* see in the jobtracker logs, suggested DNS, so I
went down that rabbit hole for quite a while.

 

Dave

 

 

From: Shahab Yunus [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, May 14, 2013 6:56 PM
To: [EMAIL PROTECTED]
Subject: Re: JobClient: Error reading task output - after instituting a DNS
server

 

HI David. an you explain in a bit more detail what was the issue? Thanks.

 

Shahab

 

On Tue, May 14, 2013 at 2:29 AM, David Parks <[EMAIL PROTECTED]> wrote:

I just hate it when I figure out a problem right after asking for help.

 

Finding the task logs via the task tracker website identified the problem
which didn't show up elsewhere. Simple mis-configuration which I did
concurrently with the DNS update that threw me off track.

 

Dave

 

 

From: David Parks [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, May 14, 2013 1:20 PM
To: [EMAIL PROTECTED]
Subject: JobClient: Error reading task output - after instituting a DNS
server

 

So we just configured a local DNS server for hostname resolution and stopped
using a hosts file and now jobs fail on us. But I can't figure out why.

 

You can see the error below, but if I run curl to any of those URLs they
come back "Failed to retrieve stdout log", which doesn't look much like a
DNS issue.

 

I can ping and do nslookup from any host to any other host. This is a CDH4
cluster and the host inspector is happy as could be; also Cloudera Manager
indicates all is well.

 

When I open the task tracker website I see the first task attempt show up on
the site there for maybe 10 seconds or so before it fails.

 

Any idea what I need to look at here?

 

Job:

===
13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process :
131

13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process :
1

13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003

13/05/14 05:13:43 INFO mapred.JobClient:  map 0% reduce 0%

13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_m_000353_0, Status : FAILED

java.lang.Throwable: Child Error

        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)

Caused by: java.io.IOException: Task process exit with nonzero status of 1.

        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)

 

13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stdout

13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stderr

13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_r_000521_0, Status : FAILED

java.lang.Throwable: Child Error

        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)

Caused by: java.io.IOException: Task process exit with nonzero status of 1.

        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)

 

13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stdout

13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stderr

 

 

curl of above URL:

===================
davidparks21@hadoop-meta1:~$ curl
'http://hadoop-fullslot2:50060/tasklog?plaintext=true
<http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_2013
05131758_0003_m_000353_0&filter=stdout>
&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout'

<html>

<head>

<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>

<title>Error 410 Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</title>

</head>

<body><h2>HTTP ERROR 410</h2>

<p>Problem accessing /tasklog. Reason:

<pre>    Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
Jetty://</small></i><br/>                                

<br/>

<br/>

<br/>

<br/>