Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> running bigger pig jobs on amazon ec2


+
jr 2010-12-08, 14:09
Copy link to this message
-
Re: running bigger pig jobs on amazon ec2
>From the logs it looks like issue is not with Pig but with your hdfs.
Either your hdfs is running out of space or some (or all) nodes in
your cluster can't talk to each other (network issue ?)

Ashutosh
On Wed, Dec 8, 2010 at 06:09, jr <[EMAIL PROTECTED]> wrote:
> Hi guys,
> I'm having some trouble finished jobs that run smoothly on a smaller
> dataset, but always fail at 99% if i try to run the job on the whole
> set.
> i can see a few killed map and a few killed reduce, but quite a lot of
> failed reduce tasks that all show the same exception at the end.
> here is what i have in the logs:
>
> 2010-12-08 08:44:56,127 INFO org.apache.hadoop.mapred.ReduceTask:
> Ignoring obsolete output of KILLED map-task:
> 'attempt_201012080810_0003_m_000009_1'
> 2010-12-08 08:45:08,152 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201012080810_0003_r_000000_0: Got 1 new map-outputs
> 2010-12-08 08:45:13,103 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201012080810_0003_r_000000_0 Scheduled 1 outputs (0 slow hosts and0 dup hosts)
> 2010-12-08 08:45:13,241 INFO org.apache.hadoop.mapred.ReduceTask: header: attempt_201012080810_0003_m_000003_0, compressed len: 3488519, decompressed len: 3488515
> 2010-12-08 08:45:13,241 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 3488515 bytes (3488519 raw bytes) into RAM from attempt_201012080810_0003_m_000003_0
> 2010-12-08 08:45:13,348 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection threshold exceeded) init = 5439488(5312K) used = 78403496(76565K) committed = 101908480(99520K) max = 139853824(136576K)
> 2010-12-08 08:45:13,404 INFO org.apache.hadoop.mapred.ReduceTask: Read 3488515 bytes from map-output for attempt_201012080810_0003_m_000003_0
> 2010-12-08 08:45:13,405 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 from attempt_201012080810_0003_m_000003_0 -> (142, 21) from ip-10-98-71-195.ec2.internal
> 2010-12-08 08:45:14,241 INFO org.apache.hadoop.mapred.ReduceTask: GetMapEventsThread exiting
> 2010-12-08 08:45:14,241 INFO org.apache.hadoop.mapred.ReduceTask: getMapsEventsThread joined.
> 2010-12-08 08:45:14,242 INFO org.apache.hadoop.mapred.ReduceTask: Closed ram manager
> 2010-12-08 08:45:14,253 INFO org.apache.hadoop.mapred.ReduceTask: Interleaved on-disk merge complete: 2 files left.
> 2010-12-08 08:45:14,254 INFO org.apache.hadoop.mapred.ReduceTask: In-memory merge complete: 64 files left.
> 2010-12-08 08:45:14,312 INFO org.apache.hadoop.mapred.Merger: Merging 64 sorted segments
> 2010-12-08 08:45:14,313 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 64 segments left of total size: 82947024 bytes
> 2010-12-08 08:45:15,389 INFO org.apache.hadoop.mapred.ReduceTask: Merged 64 segments, 82947024 bytes to disk to satisfy reduce memory limit
> 2010-12-08 08:45:15,390 INFO org.apache.hadoop.mapred.ReduceTask: Merging 3 files, 214514578 bytes from disk
> 2010-12-08 08:45:15,392 INFO org.apache.hadoop.mapred.ReduceTask: Merging 0 segments, 0 bytes from memory into reduce
> 2010-12-08 08:45:15,392 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments
> 2010-12-08 08:45:15,397 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 214514566 bytes
> 2010-12-08 08:45:15,489 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library
> 2010-12-08 08:45:15,522 INFO com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 3e7c9dcf0ea0acbde146cb22b236978b344c5525]
> 2010-12-08 08:45:15,530 INFO com.twitter.elephantbird.pig.load.LzoBaseRegexLoader: LzoBaseRegexLoader created.
> 2010-12-08 08:45:15,534 INFO com.twitter.elephantbird.pig.load.LzoBaseRegexLoader: LzoBaseRegexLoader created.
> 2010-12-08 08:45:15,544 INFO com.twitter.elephantbird.pig.load.LzoBaseRegexLoader: LzoBaseRegexLoader created.
> 2010-12-08 08:45:15,562 INFO com.twitter.elephantbird.pig.load.LzoBaseRegexLoader: LzoBaseRegexLoader created.
+
jr 2010-12-10, 10:53
+
Dmitriy Ryaboy 2010-12-12, 11:18
+
Johannes Rußek 2010-12-14, 14:47
+
Dmitriy Ryaboy 2010-12-15, 02:05