Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Reducer MapFileOutpuFormat


+
Mike S 2012-07-23, 20:09
Copy link to this message
-
Re: Reducer MapFileOutpuFormat
Hey Mike,

Inline.

On Tue, Jul 24, 2012 at 1:39 AM, Mike S <[EMAIL PROTECTED]> wrote:
> If I set my reducer output to map file output format and the job would
> say have 100 reducers, will the output generate 100 different index
> file (one for each reducer) or one index file for all the reducers
> (basically one index file per job)?

Each MapFile gets its own index file, so yes 100 index files for 100
map files, given each Reducer creating one map file as its output.

> If it is one index file per reducer, can rely on HDFS append to change
> the index write behavior and build one index file from all the
> reducers by basically making all the parallel reducers to append to
> one index file? Data files do not matter.

I don't think MapFiles are append-able yet. For one, the index
complicates things as it keeps offsets based from 0 to length (I
think). Work for easy-appending sequence files has been ongoing
though: https://issues.apache.org/jira/browse/HADOOP-7139. Maybe you
can take a look and help enable MapFiles do the same somehow?

--
Harsh J
+
syed kather 2012-07-27, 01:15
+
Bertrand Dechoux 2012-07-27, 05:54
+
Harsh J 2012-07-27, 22:07
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB