Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> possibly Pig throttles the number of mappers


Copy link to this message
-
Re: possibly Pig throttles the number of mappers
And the nodes are pretty lightly loaded (~1.0) and there's plenty of free
memory. Now I'm seeing 2 mappers per node. Very much under-utilized.

On Wed, Mar 23, 2011 at 1:39 PM, Dexin Wang <[EMAIL PROTECTED]> wrote:

> Hi,
>
> We've seen a strange problem where some Pig jobs would just run fewer
> mappers concurrently than the mapper capacity. Specifically we have a 10
> node cluster and each is configured to have 12 mappers. Normally we have 120
> mappers running. But for some Pig jobs it will only have 10 mappers running
> (while nothing else is running), and actually appears to be 1 mapper per
> node.
>
> We have not noticed the same problem with other non-Pig hadoop job. Anyone
> has experienced the same thing and have any explanation or remedy?
>
> Thanks!
> Dexin
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB