Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Map performance with custom binary format


+
william kinney 2009-07-28, 15:18
+
Scott Carey 2009-07-28, 16:58
+
Ted Dunning 2009-07-28, 18:13
+
william kinney 2009-07-28, 19:15
+
Ted Dunning 2009-07-28, 20:25
Copy link to this message
-
Re: Map performance with custom binary format
On Tue, Jul 28, 2009 at 01:25:49PM -0700, Ted Dunning wrote:
> On Tue, Jul 28, 2009 at 12:15 PM, william kinney
> <[EMAIL PROTECTED]>wrote:
>
> >
> > Also, from the job page (different job, same Map method, just more
> > data...~40GB. 781 files):
> > Map input records       629,738,080
> > Map input bytes         41,538,992,880
> >
> > Anything else I can look into?
>
>
> Yes.  The number of data local maps and how many maps total.
>

Do "data local maps" short-circuit to the local filesystem at all, or do
they read data over HTTP from the data node's jetty instance over the
loopback device?

-Erik
+
Ted Dunning 2009-07-28, 22:15
+
william kinney 2009-07-28, 21:40
+
Ted Dunning 2009-07-28, 20:27
+
Scott Carey 2009-07-28, 22:35
+
Jason Venner 2009-07-29, 02:01
+
william kinney 2009-07-30, 04:10
+
william kinney 2009-07-30, 04:07
+
Scott Carey 2009-07-30, 05:31
+
william kinney 2009-07-30, 14:37
+
william kinney 2009-07-30, 17:42
+
Scott Carey 2009-07-30, 18:39
+
Todd Lipcon 2009-07-30, 18:51
+
Scott Carey 2009-07-30, 20:32
+
Scott Carey 2009-07-30, 18:19
+
william kinney 2009-07-30, 21:32
+
Scott Carey 2009-08-01, 01:07
+
Scott Carey 2009-07-31, 21:31
+
Steve Loughran 2009-07-29, 09:17
+
Todd Lipcon 2009-07-29, 18:47
+
Scott Carey 2009-07-29, 19:07
+
Todd Lipcon 2009-07-29, 19:10
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB