Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> How to speed up the copy phrase?


Copy link to this message
-
Re: How to speed up the copy phrase?
Jothi ,

Do have any idea , how thease speeds are calculated , I mean some
mathematical expressions or stuff.

On Mon, Aug 24, 2009 at 12:54 PM, Jothi Padmanabhan
<[EMAIL PROTECTED]>wrote:

> The transfer rate is a little misleading. The timer for this calculation
> starts when the reducer itself starts and so includes the time spent by the
> reducer waiting for maps to complete. So, the speed shown when shuffling
> the
> first few maps might be totally misleading, it does not necessarily reflect
> the network speed. You should be able to see more reasonable numbers
> towards
> the end of shuffle.
>
> Jothi
>
>
> On 8/24/09 12:19 PM, "yang song" <[EMAIL PROTECTED]> wrote:
>
> > Hello, everyone
> >
> > When I submit a big job(e.g. maptasks:10000, reducetasks:500), I find
> that
> > the copy phrase will last for a long long time. From WebUI, the message
> > "reduce > copy (xxxx of 10000 at 0.01 MB/s) >" tells me the transfer
> speed
> > is just 0.01 MB/s. Does it a regular value? How can I solve it?
> >
> > Thank you!
> >
> > P.S. The hadoop version is 0.19.1. The cluster has 20 nodes. Heap size of
> JT
> > is 6G while the     others are default settings.
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB