Hello,

In hdfs we have set block size - 40bytes . Input Data set is as below
terminated with line feed.

data1   (5*8=40 bytes)
data2
......
.......
data10
 
 
But still we see only 2 map tasks spawned, should have been atleast 10 map
tasks. Each mapper performs complex mathematical computation. Not sure how
works internally. Line feed does not work. Even with below settings map
tasks never goes beyound 2, any way to make this spawn 10 tasks. Basically
it should look like compute grid - computation in parallel.
 
<property>
  <name>io.bytes.per.checksum</name>
  <value>30</value>
  <description>The number of bytes per checksum.  Must not be larger than
  io.file.buffer.size.</description>
</property>

<property>
  <name>dfs.block.size</name>
   <value>30</value>
  <description>The default block size for new files.</description>
</property>

<property>
  <name>mapred.tasktracker.map.tasks.maximum</name>
  <value>10</value>
  <description>The maximum number of map tasks that will be run
  simultaneously by a task tracker.
  </description>
</property>

single node with high configuration -> 8 cpus and 8gb memory. Hence taking
an example of 10 data items with line feeds. We want to utilize full power
of machine - hence want at least 10 map tasks - each task needs to perform
highly complex mathematical simulation.  At present it looks like file data
is the only way to specify number of map tasks via splitsize (in bytes) -
but I prefer some criteria like line feed or whatever.

How do we get 10 map tasks from above configuration - pls help.

thanks
 
--
View this message in context: http://old.nabble.com/increase-number-of-map-tasks-tp33107775p33107775.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Hi Satish
      What is your value for mapred.max.split.size? Try setting these
values as well
mapred.min.split.size=0 (it is the default value)
mapred.max.split.size=40

Try executing your job once you apply these changes on top of others you
did.

Regards
Bejoy.K.S

On Mon, Jan 9, 2012 at 10:16 PM, sset <[EMAIL PROTECTED]> wrote:

>
> Hello,
>
> In hdfs we have set block size - 40bytes . Input Data set is as below
> terminated with line feed.
>
> data1   (5*8=40 bytes)
> data2
> ......
> .......
> data10
>
>
> But still we see only 2 map tasks spawned, should have been atleast 10 map
> tasks. Each mapper performs complex mathematical computation. Not sure how
> works internally. Line feed does not work. Even with below settings map
> tasks never goes beyound 2, any way to make this spawn 10 tasks. Basically
> it should look like compute grid - computation in parallel.
>
> <property>
>  <name>io.bytes.per.checksum</name>
>  <value>30</value>
>  <description>The number of bytes per checksum.  Must not be larger than
>  io.file.buffer.size.</description>
> </property>
>
> <property>
>  <name>dfs.block.size</name>
>   <value>30</value>
>  <description>The default block size for new files.</description>
> </property>
>
> <property>
>  <name>mapred.tasktracker.map.tasks.maximum</name>
>  <value>10</value>
>  <description>The maximum number of map tasks that will be run
>  simultaneously by a task tracker.
>  </description>
> </property>
>
> single node with high configuration -> 8 cpus and 8gb memory. Hence taking
> an example of 10 data items with line feeds. We want to utilize full power
> of machine - hence want at least 10 map tasks - each task needs to perform
> highly complex mathematical simulation.  At present it looks like file data
> is the only way to specify number of map tasks via splitsize (in bytes) -
> but I prefer some criteria like line feed or whatever.
>
> How do we get 10 map tasks from above configuration - pls help.
>
> thanks
>
> --
> View this message in context:
> http://old.nabble.com/increase-number-of-map-tasks-tp33107775p33107775.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>

Hi.

I am no expert, but you could try this.

Your problem, I guess, is that the record reader reads multiple lines of
work (tasks) and gives to each mapper and thus if you only have a few tasks
(line of work in the input file) Hadoop will not spawn multiple mappers.

You could try this, make each input record, in your now single input file,
an independent file (with only one line) and give as input to your job the
the directory with the files (not a single file). For me, this forced the
spawning of multiple mappers.

There is another more correct way that forces the spawning of a map task for
each line but as I was using c++ pipes that was not an option for me.

Hope this helps
   GorGo

sset wrote:
>
> Hello,
>
> In hdfs we have set block size - 40bytes . Input Data set is as below
> terminated with line feed.
>
> data1   (5*8=40 bytes)
> data2
> ......
> .......
> data10
>  
>  
> But still we see only 2 map tasks spawned, should have been atleast 10 map
> tasks. Each mapper performs complex mathematical computation. Not sure how
> works internally. Line feed does not work. Even with below settings map
> tasks never goes beyound 2, any way to make this spawn 10 tasks. Basically
> it should look like compute grid - computation in parallel.
>  
> <property>
>   <name>io.bytes.per.checksum</name>
>   <value>30</value>
>   <description>The number of bytes per checksum.  Must not be larger than
>   io.file.buffer.size.</description>
> </property>
>
> <property>
>   <name>dfs.block.size</name>
>    <value>30</value>
>   <description>The default block size for new files.</description>
> </property>
>
> <property>
>   <name>mapred.tasktracker.map.tasks.maximum</name>
>   <value>10</value>
>   <description>The maximum number of map tasks that will be run
>   simultaneously by a task tracker.
>   </description>
> </property>
>
> single node with high configuration -> 8 cpus and 8gb memory. Hence taking
> an example of 10 data items with line feeds. We want to utilize full power
> of machine - hence want at least 10 map tasks - each task needs to perform
> highly complex mathematical simulation.  At present it looks like file
> data is the only way to specify number of map tasks via splitsize (in
> bytes) - but I prefer some criteria like line feed or whatever.
>
> How do we get 10 map tasks from above configuration - pls help.
>
> thanks
>  
>

--
View this message in context: http://old.nabble.com/increase-number-of-map-tasks-tp33107775p33111745.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Similarly there is the NLineInputFormat that does this automatically.  If your input is small it will read in the input and make a split for every N lines of input.   Then you don't have to reformat your data files.

--Bobby Evans

On 1/10/12 8:09 AM, "GorGo" <[EMAIL PROTECTED]> wrote:

Hi.

I am no expert, but you could try this.

Your problem, I guess, is that the record reader reads multiple lines of
work (tasks) and gives to each mapper and thus if you only have a few tasks
(line of work in the input file) Hadoop will not spawn multiple mappers.

You could try this, make each input record, in your now single input file,
an independent file (with only one line) and give as input to your job the
the directory with the files (not a single file). For me, this forced the
spawning of multiple mappers.

There is another more correct way that forces the spawning of a map task for
each line but as I was using c++ pipes that was not an option for me.

Hope this helps
   GorGo

sset wrote:
>
> Hello,
>
> In hdfs we have set block size - 40bytes . Input Data set is as below
> terminated with line feed.
>
> data1   (5*8=40 bytes)
> data2
> ......
> .......
> data10
>
>
> But still we see only 2 map tasks spawned, should have been atleast 10 map
> tasks. Each mapper performs complex mathematical computation. Not sure how
> works internally. Line feed does not work. Even with below settings map
> tasks never goes beyound 2, any way to make this spawn 10 tasks. Basically
> it should look like compute grid - computation in parallel.
>
> <property>
>   <name>io.bytes.per.checksum</name>
>   <value>30</value>
>   <description>The number of bytes per checksum.  Must not be larger than
>   io.file.buffer.size.</description>
> </property>
>
> <property>
>   <name>dfs.block.size</name>
>    <value>30</value>
>   <description>The default block size for new files.</description>
> </property>
>
> <property>
>   <name>mapred.tasktracker.map.tasks.maximum</name>
>   <value>10</value>
>   <description>The maximum number of map tasks that will be run
>   simultaneously by a task tracker.
>   </description>
> </property>
>
> single node with high configuration -> 8 cpus and 8gb memory. Hence taking
> an example of 10 data items with line feeds. We want to utilize full power
> of machine - hence want at least 10 map tasks - each task needs to perform
> highly complex mathematical simulation.  At present it looks like file
> data is the only way to specify number of map tasks via splitsize (in
> bytes) - but I prefer some criteria like line feed or whatever.
>
> How do we get 10 map tasks from above configuration - pls help.
>
> thanks
>
>

--
View this message in context: http://old.nabble.com/increase-number-of-map-tasks-tp33107775p33111745.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Thanks. Seperate files for line item has created 10 map tasks out of which
only some are in running state (given by max map reduce tasks)  rest are in
wait. So if I have 8 cpus, I have max_map_tasks as 7 so 3 are in wait state.
I can see 7 cpus utilization 90-95%.

1. Performance tunning/optimization any good suggestions or links?

2. Logging - If I do any logging in map/reduce class where will be logging
or system.out information written?

3. How do we reuse jvm? map tasks creation takes time.

4. Different types of spills - how do we avoid them?

--
View this message in context: http://old.nabble.com/increase-number-of-map-tasks-tp33107775p33128748.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

1. Performance tunning/optimization any good suggestions or links?
    Take a look at
http://wiki.datameer.com/documentation/current/Hadoop+Cluster+Configuration+Tips

2. Logging - If I do any logging in map/reduce class where will be logging
or system.out information written?
Be careful while doing so since on large amounts of data you can fill up
disk on datanodes very quickly. You can find the logs through the
jobtracker page by clicking on specific map and reduce tasks.

3. How do we reuse jvm? map tasks creation takes time.
Look at mapred.job.reuse.jvm.num.tasks

4. Different types of spills - how do we avoid them?
Depends on what is causing the spills. You can have spills on Map and
Reduce side, and adjusting config properties such "io.sort.mb",
"io.sort.factor", and a few others on the Reduce side. Tom White's book has
a good explanation on these.

Thanks,
Prashant Kommireddi
On Thu, Jan 12, 2012 at 8:10 AM, screen <[EMAIL PROTECTED]> wrote:

>
> Thanks. Seperate files for line item has created 10 map tasks out of which
> only some are in running state (given by max map reduce tasks)  rest are in
> wait. So if I have 8 cpus, I have max_map_tasks as 7 so 3 are in wait
> state.
> I can see 7 cpus utilization 90-95%.
>
> 1. Performance tunning/optimization any good suggestions or links?
>
> 2. Logging - If I do any logging in map/reduce class where will be logging
> or system.out information written?
>
> 3. How do we reuse jvm? map tasks creation takes time.
>
> 4. Different types of spills - how do we avoid them?
>
> --
> View this message in context:
> http://old.nabble.com/increase-number-of-map-tasks-tp33107775p33128748.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>

Hi.

I may not be the best person to answer this as my mapper is not written in
JAVA but here are my 2 cents worth.

screen wrote:
>
> 1. Performance tunning/optimization any good suggestions or links?
>
With only 10 tasks, and a heavy calculation in the map() tuning the hadoop
is not your first priority.
However, if you plan to make more tasks on cluster of machines performance
still depends on your code data and I/O profile etc.
I recommend the book, "Hadoop - The Definitive Guide" published by O'Reilly.
It does not answer all my questions but it is handy to have on  your
desktop.
screen wrote:
>
> 2. Logging - If I do any logging in map/reduce class where will be logging
> or system.out information written?
>
I am no expert here, but you give the job a working directory in the job
configuration file, I think you will find the logs there.
 

screen wrote:
>
> 3. How do we reuse jvm? map tasks creation takes time.
>
The overhead of starting a JVM is about a second on the Hadoop side and will
only benefit you if you have many very short  tasks, or if you have lengthy
initialization on your side.
"mapred.job.reuse.jvm.num.tasks can be used to specify the maximum number of
tasks to run for a given job for each JVM " (from the book recommended
above). -1 sets it to no limit.
It can also be set on a per job basis with setNumTasksToExecutePerJvm().
Again, I recommend the book.
screen wrote:
>
> 4. Different types of spills - how do we avoid them?
>
Not sure exactly what you mean here. In general it is best to have many
short tasks then to manually pick the map tasks and segment so it fits to
CPUS.
Why, well if something goes wrong (lets say one CPU acts up or is busy doing
something else) your manual plan fails. Second, the number of map tasks you
set should be set to much higher as that is just the maximum and only a hint
to Hadoop on how many to spawn. You do not control the map spawning.
If you want such accurate control I would think doing the work in the
reducer would be better as you do control how many they are.
Also, you could consider speculative execution, i.e. allowing two CPUs to do
the same work at the same time. This is useful if you have workers that are
not keeping up, and in the end there are some idle workers waiting for the
last work to be done.
In your case you have 10 tasks, and 7 workers .. lets say that for some
reason 1 of the cpus is much slower than the others, then speculative
execution could have all free cpus do the last 3 tasks and even start the
ones that have not finished of the first 7, just in case they are very very
slow.
However this is usually much more useful in multi machine cases where the
diff. can be inherent (not the same hardware etc.)

hope you find some of this useful.
Regards
    Gorgo
--
View this message in context: http://old.nabble.com/increase-number-of-map-tasks-tp33107775p33132789.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.