Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - possibly Pig throttles the number of mappers


Copy link to this message
-
Re: possibly Pig throttles the number of mappers
Dexin Wang 2011-03-24, 00:45
And the nodes are pretty lightly loaded (~1.0) and there's plenty of free
memory. Now I'm seeing 2 mappers per node. Very much under-utilized.

On Wed, Mar 23, 2011 at 1:39 PM, Dexin Wang <[EMAIL PROTECTED]> wrote:

> Hi,
>
> We've seen a strange problem where some Pig jobs would just run fewer
> mappers concurrently than the mapper capacity. Specifically we have a 10
> node cluster and each is configured to have 12 mappers. Normally we have 120
> mappers running. But for some Pig jobs it will only have 10 mappers running
> (while nothing else is running), and actually appears to be 1 mapper per
> node.
>
> We have not noticed the same problem with other non-Pig hadoop job. Anyone
> has experienced the same thing and have any explanation or remedy?
>
> Thanks!
> Dexin
>