-Re: Can MapReduce run simultaneous producer/consumer processes?
Rahul Jain 2011-01-07, 00:42
In case the producer / consumer don't require sorting to happen, take a look
If you do want the stuff to happen after sorting, take a look at:
More esoteric cases will require you to do separate map reduces, at least
with the current hadoop framework.
On Thu, Jan 6, 2011 at 2:27 PM, W.P. McNeill <[EMAIL PROTECTED]> wrote:
> Say I have two MapReduce processes, A and B. The two are algorithmically
> dissimilar, so they have to be implemented as separate MapReduce processes.
> The output of A is used as the input of B, so A has to run first.
> B doesn't need to take all of A's output as input, only a partition of it.
> So in theory A and B could run at the same time in a producer/consumer
> arrangement, where B would start to work as soon as A had produced some
> output but before A had completed. Obviously, this could be a big
> parallelization win.
> Is this possible in MapReduce? I know at the most basic level it is
> not–there is no synchronization mechanism that allows the same HDFS
> directory to be used for both input and output–but is there some
> layer on top that allows it? I've been digging around, and I think the
> answer is "No" but I want to be sure.
> More specifically, the only abstraction layer I'm aware of that chains
> together MapReduce processes is Cascade, and I think it requires the reduce
> steps to be serialized, but again I'm not sure because I've only read the
> documentation and haven't actually played with it.