Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> How to speed up the copy phrase?


Copy link to this message
-
Re: How to speed up the copy phrase?
Jothi ,

Do have any idea , how thease speeds are calculated , I mean some
mathematical expressions or stuff.

On Mon, Aug 24, 2009 at 12:54 PM, Jothi Padmanabhan
<[EMAIL PROTECTED]>wrote:

> The transfer rate is a little misleading. The timer for this calculation
> starts when the reducer itself starts and so includes the time spent by the
> reducer waiting for maps to complete. So, the speed shown when shuffling
> the
> first few maps might be totally misleading, it does not necessarily reflect
> the network speed. You should be able to see more reasonable numbers
> towards
> the end of shuffle.
>
> Jothi
>
>
> On 8/24/09 12:19 PM, "yang song" <[EMAIL PROTECTED]> wrote:
>
> > Hello, everyone
> >
> > When I submit a big job(e.g. maptasks:10000, reducetasks:500), I find
> that
> > the copy phrase will last for a long long time. From WebUI, the message
> > "reduce > copy (xxxx of 10000 at 0.01 MB/s) >" tells me the transfer
> speed
> > is just 0.01 MB/s. Does it a regular value? How can I solve it?
> >
> > Thank you!
> >
> > P.S. The hadoop version is 0.19.1. The cluster has 20 nodes. Heap size of
> JT
> > is 6G while the     others are default settings.
>
>