Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> issue about how to assiging map output to reducer?


Copy link to this message
-
Re: issue about how to assiging map output to reducer?
Depends on the distribution of the keys and how the partitioner is
assigning keys to reducers.  (Remember that pairs with the same key have to
go to the same reducer).
Chris
On Jan 8, 2014 2:33 AM, "ch huang" <[EMAIL PROTECTED]> wrote:

> hi,maillist:
>             i look the containers log from " hadoop fs -cat
> /var/log/hadoop-yarn/apps/root/logs/application_1388730279827_2770/CHBM221_50853"
>
> and log say it get 25 map output , and assiging 7 to fetcher 5, assiging 7
> to fetcher 4 and assiging 11 to fetcher 3,my question is why not
> " assiging 8 to fetcher 5, assiging 8 to fetcher 4 and assiging 9 to
> fetcher 3 " ?
>
> 2014-01-08 11:28:00,346 INFO [EventFetcher for fetching Map Completion
> Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher:
> attempt_1388730279827_2770_r_000000_0: Got 25 new map-outputs
> 2014-01-08 11:28:00,348 INFO [fetcher#5]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM223:8080 with 7 to fetcher#5
> 2014-01-08 11:28:00,349 INFO [fetcher#5]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7
> to CHBM223:8080 to fetcher#5
> 2014-01-08 11:28:00,349 INFO [fetcher#4]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM222:8080 with 7 to fetcher#4
> 2014-01-08 11:28:00,349 INFO [fetcher#4]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7
> to CHBM222:8080 to fetcher#4
> 2014-01-08 11:28:00,352 INFO [fetcher#3]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM221:8080 with 11 to fetcher#3
> 2014-01-08 11:28:00,352 INFO [fetcher#3]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 11 of 11
> to CHBM221:8080 to fetcher#3
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB