Hello, everyone

When I submit a big job(e.g. maptasks:10000, reducetasks:500), I find that
the copy phrase will last for a long long time. From WebUI, the message
"reduce > copy (xxxx of 10000 at 0.01 MB/s) >" tells me the transfer speed
is just 0.01 MB/s. Does it a regular value? How can I solve it?

Thank you!

P.S. The hadoop version is 0.19.1. The cluster has 20 nodes. Heap size of JT
is 6G while the     others are default settings.

The transfer rate is a little misleading. The timer for this calculation
starts when the reducer itself starts and so includes the time spent by the
reducer waiting for maps to complete. So, the speed shown when shuffling the
first few maps might be totally misleading, it does not necessarily reflect
the network speed. You should be able to see more reasonable numbers towards
the end of shuffle.

Jothi
On 8/24/09 12:19 PM, "yang song" <[EMAIL PROTECTED]> wrote:

> Hello, everyone
>
> When I submit a big job(e.g. maptasks:10000, reducetasks:500), I find that
> the copy phrase will last for a long long time. From WebUI, the message
> "reduce > copy (xxxx of 10000 at 0.01 MB/s) >" tells me the transfer speed
> is just 0.01 MB/s. Does it a regular value? How can I solve it?
>
> Thank you!
>
> P.S. The hadoop version is 0.19.1. The cluster has 20 nodes. Heap size of JT
> is 6G while the     others are default settings.

Maybe look at mapred.reduce.parallel.copies property to speed it up...I don't see as to why transfer speed be configured via params, and I'm think hadoop wont be messing with that.

Thanks,
Amogh

-----Original Message-----
From: yang song [mailto:[EMAIL PROTECTED]]
Sent: Monday, August 24, 2009 12:20 PM
To: [EMAIL PROTECTED]
Subject: How to speed up the copy phrase?

Hello, everyone

When I submit a big job(e.g. maptasks:10000, reducetasks:500), I find that
the copy phrase will last for a long long time. From WebUI, the message
"reduce > copy (xxxx of 10000 at 0.01 MB/s) >" tells me the transfer speed
is just 0.01 MB/s. Does it a regular value? How can I solve it?

Thank you!

P.S. The hadoop version is 0.19.1. The cluster has 20 nodes. Heap size of JT
is 6G while the     others are default settings.

If you've got 20 nodes, then you want to have 20-ish reduce tasks. Maybe 40
if you want it to run in two waves. (Assuming 1 core/node. Multiply by N for
N cores...) As it is, each node has 500-ish map tasks that it has to read
from and for each of these, it needs to generate 500 separate reduce task
output files.  That's going to take Hadoop a long time to do. 10000 map
tasks is also a very large number of map tasks. Are you processing a lot of
little files? If so, try using MultiFileInputFormat or MultipleInputs to
group them together.

As is mentioned, also set mapred.reduce.parallel.copies to 20. (The default
of 5 is appropriate only for 1--5 nodes.)

- Aaron

On Mon, Aug 24, 2009 at 12:31 AM, Amogh Vasekar <[EMAIL PROTECTED]> wrote:

> Maybe look at mapred.reduce.parallel.copies property to speed it up...I
> don't see as to why transfer speed be configured via params, and I'm think
> hadoop wont be messing with that.
>
> Thanks,
> Amogh
>
> -----Original Message-----
> From: yang song [mailto:[EMAIL PROTECTED]]
> Sent: Monday, August 24, 2009 12:20 PM
> To: [EMAIL PROTECTED]
> Subject: How to speed up the copy phrase?
>
> Hello, everyone
>
> When I submit a big job(e.g. maptasks:10000, reducetasks:500), I find that
> the copy phrase will last for a long long time. From WebUI, the message
> "reduce > copy (xxxx of 10000 at 0.01 MB/s) >" tells me the transfer speed
> is just 0.01 MB/s. Does it a regular value? How can I solve it?
>
> Thank you!
>
> P.S. The hadoop version is 0.19.1. The cluster has 20 nodes. Heap size of
> JT
> is 6G while the     others are default settings.
>

Jothi ,

Do have any idea , how thease speeds are calculated , I mean some
mathematical expressions or stuff.

On Mon, Aug 24, 2009 at 12:54 PM, Jothi Padmanabhan
<[EMAIL PROTECTED]>wrote:

> The transfer rate is a little misleading. The timer for this calculation
> starts when the reducer itself starts and so includes the time spent by the
> reducer waiting for maps to complete. So, the speed shown when shuffling
> the
> first few maps might be totally misleading, it does not necessarily reflect
> the network speed. You should be able to see more reasonable numbers
> towards
> the end of shuffle.
>
> Jothi
>
>
> On 8/24/09 12:19 PM, "yang song" <[EMAIL PROTECTED]> wrote:
>
> > Hello, everyone
> >
> > When I submit a big job(e.g. maptasks:10000, reducetasks:500), I find
> that
> > the copy phrase will last for a long long time. From WebUI, the message
> > "reduce > copy (xxxx of 10000 at 0.01 MB/s) >" tells me the transfer
> speed
> > is just 0.01 MB/s. Does it a regular value? How can I solve it?
> >
> > Thank you!
> >
> > P.S. The hadoop version is 0.19.1. The cluster has 20 nodes. Heap size of
> JT
> > is 6G while the     others are default settings.
>
>

It is fairly straight forward, on completion of a successful fetch, the
total amount of bytes fetched is divided by the total time taken till then.

Please look at fetchOutputs method in ReduceTask.java, the portion of code
that handles successful copies.

Jothi
On 8/25/09 8:23 AM, "bharath vissapragada"
<[EMAIL PROTECTED]> wrote:

> Jothi ,
>
> Do have any idea , how thease speeds are calculated , I mean some
> mathematical expressions or stuff.
>
> On Mon, Aug 24, 2009 at 12:54 PM, Jothi Padmanabhan
> <[EMAIL PROTECTED]>wrote:
>
>> The transfer rate is a little misleading. The timer for this calculation
>> starts when the reducer itself starts and so includes the time spent by the
>> reducer waiting for maps to complete. So, the speed shown when shuffling
>> the
>> first few maps might be totally misleading, it does not necessarily reflect
>> the network speed. You should be able to see more reasonable numbers
>> towards
>> the end of shuffle.
>>
>> Jothi
>>
>>
>> On 8/24/09 12:19 PM, "yang song" <[EMAIL PROTECTED]> wrote:
>>
>>> Hello, everyone
>>>
>>> When I submit a big job(e.g. maptasks:10000, reducetasks:500), I find
>> that
>>> the copy phrase will last for a long long time. From WebUI, the message
>>> "reduce > copy (xxxx of 10000 at 0.01 MB/s) >" tells me the transfer
>> speed
>>> is just 0.01 MB/s. Does it a regular value? How can I solve it?
>>>
>>> Thank you!
>>>
>>> P.S. The hadoop version is 0.19.1. The cluster has 20 nodes. Heap size of
>> JT
>>> is 6G while the     others are default settings.
>>
>>

On Aug 24, 2009, at 5:49 PM, Aaron Kimball wrote:

> If you've got 20 nodes, then you want to have 20-ish reduce tasks.  
> Maybe 40
> if you want it to run in two waves. (Assuming 1 core/node. Multiply  
> by N for
> N cores...) As it is, each node has 500-ish map tasks that it has to  
> read
> from and for each of these, it needs to generate 500 separate reduce  
> task
> output files.  That's going to take Hadoop a long time to do.

Maps do not produce one output file per reduce, the entire map-output  
is in a single file.

Arun

Interesting.  In this case, how does Jetty dole out the proper
partitions of the intermediate data to the appropriate reducers if
they are located in the same files?

Thanks,
George

On Thu, Aug 27, 2009 at 11:31 AM, Arun C Murthy<[EMAIL PROTECTED]> wrote:
>
> On Aug 24, 2009, at 5:49 PM, Aaron Kimball wrote:
>
>> If you've got 20 nodes, then you want to have 20-ish reduce tasks. Maybe
>> 40
>> if you want it to run in two waves. (Assuming 1 core/node. Multiply by N
>> for
>> N cores...) As it is, each node has 500-ish map tasks that it has to read
>> from and for each of these, it needs to generate 500 separate reduce task
>> output files.  That's going to take Hadoop a long time to do.
>
> Maps do not produce one output file per reduce, the entire map-output is in
> a single file.
>
> Arun
>

There is an index with the offset of each reduce's first byte. The  
index is written to disk, but is also cached by the task tracker.

-- Owen

On Aug 27, 2009, at 17:12, George Porter <[EMAIL PROTECTED]> wrote:

> Interesting.  In this case, how does Jetty dole out the proper
> partitions of the intermediate data to the appropriate reducers if
> they are located in the same files?
>
> Thanks,
> George
>
> On Thu, Aug 27, 2009 at 11:31 AM, Arun C Murthy<[EMAIL PROTECTED]>  
> wrote:
>>
>> On Aug 24, 2009, at 5:49 PM, Aaron Kimball wrote:
>>
>>> If you've got 20 nodes, then you want to have 20-ish reduce tasks.  
>>> Maybe
>>> 40
>>> if you want it to run in two waves. (Assuming 1 core/node.  
>>> Multiply by N
>>> for
>>> N cores...) As it is, each node has 500-ish map tasks that it has  
>>> to read
>>> from and for each of these, it needs to generate 500 separate  
>>> reduce task
>>> output files.  That's going to take Hadoop a long time to do.
>>
>> Maps do not produce one output file per reduce, the entire map-
>> output is in
>> a single file.
>>
>> Arun
>>