-Re: Will hadoop always spread the work evenly between nodes?
jeremy p 2013-03-19, 20:49
Thank you, Jeff. Actually, it turns out the scenario you outlined is much
closer to the situation I'm dealing with. I'm going to ask a separate
question about that.
On Wed, Mar 13, 2013 at 1:47 PM, Jeffrey Buell <[EMAIL PROTECTED]> wrote:
> I think in your case it will have to be even, because all the slots will
> get filled. A more interesting case is if you have 40 nodes, will you get
> exactly 5 slots used for each of the nodes? Or will some nodes get more
> than 5 mappers, and others less? I don't remember the details, but I've
> had problems with unevenness in such scenarios. At least in MR1, you can
> usually force evenness by adjusting the number of map and reduce slots per
> node. In MR2 the slots are combined so achieving evenness will be more
> *From: *"jeremy p" <[EMAIL PROTECTED]>
> *To: *[EMAIL PROTECTED]
> *Sent: *Wednesday, March 13, 2013 1:01:46 PM
> *Subject: *Will hadoop always spread the work evenly between nodes?
> Say I have 200 input files and 20 nodes, and each node has 10 mapper
> slots. Will Hadoop always allocate the work evenly, such that each node
> will get 10 input files and simultaneously start 10 mappers? Is there a
> way to force this behavior?