Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Map performance with custom binary format


Copy link to this message
-
Re: Map performance with custom binary format
On Tue, Jul 28, 2009 at 01:25:49PM -0700, Ted Dunning wrote:
> On Tue, Jul 28, 2009 at 12:15 PM, william kinney
> <[EMAIL PROTECTED]>wrote:
>
> >
> > Also, from the job page (different job, same Map method, just more
> > data...~40GB. 781 files):
> > Map input records       629,738,080
> > Map input bytes         41,538,992,880
> >
> > Anything else I can look into?
>
>
> Yes.  The number of data local maps and how many maps total.
>

Do "data local maps" short-circuit to the local filesystem at all, or do
they read data over HTTP from the data node's jetty instance over the
loopback device?

-Erik
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB