-Re: How does map-merge work exactly?
Chris Douglas 2012-09-17, 08:21
On Thu, Sep 13, 2012 at 7:04 AM, Martin Dobmeier
<[EMAIL PROTECTED]> wrote:
> What exactly is a segment? Is it the number of spills?
A segment in this context is a fraction of spill output for a
particular reduce. Each spill contains a segment for every reduce.
> What does "0 segments left" mean? Does it mean that the merge could be
> performed on the first pass?
> Why are only 54 segments merged instead of "io.sort.factor" segments?
The intermediate merge of 54 files to 1 reduces the number of files to
117 - 53 = 64 segments. The final merge is over 64 segments.
> (io.sort.factor determines the number of files to merge during a pass,
> Why is the merge performed "number of reducers" times? (I'm counting the
> phrase "Merging 117 segments" exactly 96 times)
Each invocation of the merger is combining all the output assigned to
a reduce by the partitioner. -C