Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Mapper runs only on one machine


Copy link to this message
-
Re: Mapper runs only on one machine
Are you sure your input file is splittable - many files (say gzip) are not
and such files must be processed on a single machine

On Tue, Nov 16, 2010 at 9:24 AM, <[EMAIL PROTECTED]> wrote:

>  Hi all,
> I have been trying to figure out why all mappers run only on one machine
> when I have 4 node cluster. Ruduce part is running fine on all 4 nodes
> correctly. I am using 0.20.2. My input file is a large single file (10GB)
>
> Here is my config in mapred-site.xml. I specified map.tasks as 30 but I
> only se one map task and that too only on one machine. Are there any other
> parameters I need to set in order to control uniform distribution of map
> job?
> <configuration>
>         <property>
>           <name>mapred.job.tracker</name>
>            <value>master-hadoop:54311</value>
>           <description>The host and port that the MapReduce job tracker
> runs
>           at.  If "local", then jobs are run in-process as a single map
>           and reduce task.
>           </description>
>         </property>
>         <property>
>           <name>mapred.child.java.opts</name>
>           <value>-Xmx4096m</value>
>           <description>map heap size for child task</description>
>         </property>
>         <property>
>           <name>mapred.reduce.parallel.copies</name>
>           <value>5</value>
>           <description></description>
>         </property>
>         <property>
>           <name>mapred.map.tasks</name>
>           <value>30</value>
>           <description></description>
>         </property>
>         <property>
>           <name>mapred.reduce.tasks</name>
>           <value>6</value>
>           <description></description>
>         </property>
> </configuration>
>
>

--
Steven M. Lewis PhD
4221 105th Ave Ne
Kirkland, WA 98033
206-384-1340 (cell)
Institute for Systems Biology
Seattle WA