Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> possibly Pig throttles the number of mappers

Copy link to this message
Re: possibly Pig throttles the number of mappers
And the nodes are pretty lightly loaded (~1.0) and there's plenty of free
memory. Now I'm seeing 2 mappers per node. Very much under-utilized.

On Wed, Mar 23, 2011 at 1:39 PM, Dexin Wang <[EMAIL PROTECTED]> wrote:

> Hi,
> We've seen a strange problem where some Pig jobs would just run fewer
> mappers concurrently than the mapper capacity. Specifically we have a 10
> node cluster and each is configured to have 12 mappers. Normally we have 120
> mappers running. But for some Pig jobs it will only have 10 mappers running
> (while nothing else is running), and actually appears to be 1 mapper per
> node.
> We have not noticed the same problem with other non-Pig hadoop job. Anyone
> has experienced the same thing and have any explanation or remedy?
> Thanks!
> Dexin