Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Help. Strange thing. It's block me 1 week....


+
yaotian 2013-01-18, 03:17
Copy link to this message
-
Re: Help. Strange thing. It's block me 1 week....
What are your number of map and reduce slots configured to, per node? Also
noticed you seem to be requesting 4 GB memory from Reducers when your
slaves' maximum RAM itself nears that - the result may not be so good here
and can certainly cause slowdowns (due to swapping/etc.).
On Fri, Jan 18, 2013 at 8:47 AM, yaotian <[EMAIL PROTECTED]> wrote:

> Hi,
>
> *=>My machine environment:*
> 1 master 1 CPU core, 2G Mhz, 1G Memory
> 2 Slaves(datanode): 1 CPU core, 2G Mhz, 4G memory
> hadoop: hadoop-0.20.205.0
>
> *=> My data:*
> User GPS trace analysis. Each user has many gp location information. We
> want to analyze them.
>
> *=>My question:*
> 1. We have 2 datanode. But hadoop only used only 1 server? Is that not
> effective?
>
> 2. When i run the 200M size data. It is successful. But if i run 30G data,
> it always to report "Task attempt_201301171429_0013_r_000000_0 failed to
> report status for 600 seconds. Killing!"
>
> *=>My map-reduce config:*
> <configuration>
>
>      <property>
>          <name>mapred.job.tracker</name>
>          <value>master:9001</value>
>      </property>
>
>
> <property>
>   <name>mapred.reduce.parallel.copies</name>
>   <value>50</value>
> </property>
>
> <property>
>   <name>mapred.compress.map.output</name>
>   <value>true</value>
> </property>
>
>
> <property>
>   <name>mapred.job.shuffle.merge.percent</name>
>   <value>0.75</value>
> </property>
>
>
> <property>
>   <name>mapred.job.tracker.http.address</name>
>   <value>0.0.0.0:9003</value>
> </property>
>
>
> <property>
>   <name>mapreduce.reduce.memory.mb</name>
>   <value>4000</value>
> </property>
>
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx2000m</value>
> </property>
>
> <property>
>   <name>mapreduce.reduce.java.opts</name>
>   <value>-Xmx2000m</value>
> </property>
>
>
> <property>
>   <name>mapred.reduce.tasks</name>
>   <value>AutoReduce</value>
> </property>
>
> <property>
>   <name>io.sort.factor</name>
>   <value>12</value>
> </property>
>
> <property>
>   <name>io.sort.mb</name>
>   <value>300</value>
> </property>
>
>
> <property>
>   <name>io.file.buffer.size</name>
>   <value>65536</value>
> </property>
>
> <property>
>   <name>dfs.datanode.handler.count</name>
>   <value>8</value>
> </property>
>  </configuration>
>
>
>
>
>
>
>
--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB