Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Map performance with custom binary format


Copy link to this message
-
Re: Map performance with custom binary format
On Tue, Jul 28, 2009 at 01:25:49PM -0700, Ted Dunning wrote:
> On Tue, Jul 28, 2009 at 12:15 PM, william kinney
> <[EMAIL PROTECTED]>wrote:
>
> >
> > Also, from the job page (different job, same Map method, just more
> > data...~40GB. 781 files):
> > Map input records       629,738,080
> > Map input bytes         41,538,992,880
> >
> > Anything else I can look into?
>
>
> Yes.  The number of data local maps and how many maps total.
>

Do "data local maps" short-circuit to the local filesystem at all, or do
they read data over HTTP from the data node's jetty instance over the
loopback device?

-Erik