-Re: How to speed up the copy phrase?
bharath vissapragada 2009-08-25, 02:53
Do have any idea , how thease speeds are calculated , I mean some
mathematical expressions or stuff.
On Mon, Aug 24, 2009 at 12:54 PM, Jothi Padmanabhan
> The transfer rate is a little misleading. The timer for this calculation
> starts when the reducer itself starts and so includes the time spent by the
> reducer waiting for maps to complete. So, the speed shown when shuffling
> first few maps might be totally misleading, it does not necessarily reflect
> the network speed. You should be able to see more reasonable numbers
> the end of shuffle.
> On 8/24/09 12:19 PM, "yang song" <[EMAIL PROTECTED]> wrote:
> > Hello, everyone
> > When I submit a big job(e.g. maptasks:10000, reducetasks:500), I find
> > the copy phrase will last for a long long time. From WebUI, the message
> > "reduce > copy (xxxx of 10000 at 0.01 MB/s) >" tells me the transfer
> > is just 0.01 MB/s. Does it a regular value? How can I solve it?
> > Thank you!
> > P.S. The hadoop version is 0.19.1. The cluster has 20 nodes. Heap size of
> > is 6G while the others are default settings.