Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> issure about MR job on yarn framework


Copy link to this message
-
Re: issure about MR job on yarn framework
another question is why the map process progress will back when it reach
100%?
On Tue, Dec 3, 2013 at 10:07 AM, ch huang <[EMAIL PROTECTED]> wrote:

> hi,maillist:
>               i run a job on my CDH4.4 yarn framework ,it's map task
> finished very fast,but reduce is very slow, i check it use ps command
> find it's work heap size is 200m,so i try to increase heap size used by
> reduce task,i add "YARN_OPTS="$YARN_OPTS
> -Dmapreduce.reduce.java.opts=-Xmx1024m -verbose:gc -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps
> -Xloggc:$YARN_LOG_DIR/gc-$(hostname)-resourcemanager.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=15M
> -XX:-UseGCOverheadLimit"    in yarn-env.sh file ,but when i restart the
> nodemanager ,i find new reduce task still use 200m heap ,why?
>
> # jps
> 2853 DataNode
> 19533 Jps
> 10949 YarnChild
> 10661 NodeManager
> 15130 HRegionServer
> # ps -ef|grep 10949
> yarn     10949 10661 99 09:52 ?        00:19:31
> /usr/java/jdk1.7.0_45/bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx200m
> -Djava.io.tmpdir=/data/1/mrlocal/yarn/local/usercache/hdfs/appcache/application_1385983958793_0022/container_1385983958793_0022_01_005650/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.mapreduce.container.log.dir=/data/2/mrlocal/yarn/logs/application_1385983958793_0022/container_1385983958793_0022_01_005650
> -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 192.168.11.10 48936
> attempt_1385983958793_0022_r_000000_14 5650
>
>
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB