Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> join operation fails on big data set


+
Mua Ban 2013-04-12, 15:18
+
Cheolsoo Park 2013-04-12, 15:29
+
Mua Ban 2013-04-12, 17:27
+
Cheolsoo Park 2013-04-12, 18:25
+
Mua Ban 2013-04-12, 19:06
Copy link to this message
-
Re: join operation fails on big data set
seems a HDFS issue, as you said, cannot retrieval certain block from
certain DN. Can you check the health of all DN? And properly also bump the
log4j level to DEBUG.

Johnny
On Fri, Apr 12, 2013 at 12:06 PM, Mua Ban <[EMAIL PROTECTED]> wrote:

> Thank you very much Cheolsoo,
>
> I am running the script once more right now and I see 7 failed reducers at
> the moment on the job tracker GUI. I browse these failed reducers and I
> found the task logs. From these 7 failed reducers, some have type 1 task
> log, the rest have type 2 task log as I show below.
>
> They seem related to some connection issue among nodes in the cluster. Do
> you know any parameters I should configure to figure out the actual
> problem?
>
> Thank you,
> -Mua
>
> ---------------------------------------
> *Type 1 task log*
>
> 3-04-12 13:42:24,960 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201304081613_0049_r_000009_0 Scheduled 5 outputs (0 slow hosts and0
> dup hosts)
> 2013-04-12 13:42:25,259 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201304081613_0049_r_000009_0 Scheduled 1 outputs (0 slow hosts and0
> dup hosts)
> 2013-04-12 13:42:25,271 INFO org.apache.hadoop.mapred.ReduceTask:
> Initiating in-memory merge with 610 segments...
> 2013-04-12 13:42:25,273 INFO org.apache.hadoop.mapred.Merger: Merging 610
> sorted segments
> 2013-04-12 13:42:25,275 INFO org.apache.hadoop.mapred.Merger: Down to the
> last merge-pass, with 610 segments left of total size: 96922927 bytes
> 2013-04-12 13:42:27,348 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201304081613_0049_r_000009_0 Merge of the 610 files in-memory
> complete. Local file is
>
> /hdfs/sp/filesystem/mapred/local/taskTracker/vul/jobcache/job_201304081613_0049/attempt_201304081613_0049_r_000009_0/output/map_6.out
> of size 96921713
> 2013-04-12 13:42:27,349 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201304081613_0049_r_000009_0 Thread waiting: Thread for merging
> on-disk files
> 2013-04-12 13:42:30,263 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201304081613_0049_r_000009_0 Scheduled 1 outputs (0 slow hosts and0
> dup hosts)
> 2013-04-12 13:42:35,267 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201304081613_0049_r_000009_0 Scheduled 2 outputs (0 slow hosts and0
> dup hosts)
> 2013-04-12 13:42:38,145 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring
> obsolete output of KILLED map-task: 'attempt_201304081613_0049_m_000584_0'
> 2013-04-12 13:42:44,150 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring
> obsolete output of KILLED map-task: 'attempt_201304081613_0049_m_000557_0'
> 2013-04-12 13:42:55,283 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201304081613_0049_r_000009_0 Scheduled 1 outputs (0 slow hosts and0
> dup hosts)
> 2013-04-12 13:43:05,164 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring
> obsolete output of KILLED map-task: 'attempt_201304081613_0049_m_000604_0'
> 2013-04-12 13:43:06,036 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201304081613_0049_r_000009_0 Scheduled 1 outputs (0 slow hosts and0
> dup hosts)
> 2013-04-12 13:43:11,169 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring
> obsolete output of KILLED map-task: 'attempt_201304081613_0049_m_000597_1'
> 2013-04-12 13:43:21,040 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201304081613_0049_r_000009_0 Need another 5 map output(s) where 0
> is already in progress
> 2013-04-12 13:43:21,040 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201304081613_0049_r_000009_0 Scheduled 0 outputs (0 slow hosts and0
> dup hosts)
> 2013-04-12 13:44:21,042 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201304081613_0049_r_000009_0 Need another 5 map output(s) where 0
> is already in progress
> 2013-04-12 13:44:21,043 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_201304081613_0049_r_000009_0 Scheduled 1 outputs (0 slow hosts and0
> dup hosts)
> 2013-04-12 13:44:29,222 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring
> obsolete output of KILLED map-task: 'attempt_201304081613_0049_m_000576_0'
+
Mua Ban 2013-04-14, 13:13
+
Johnny Zhang 2013-04-15, 17:48