Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Intermediate files generated.


Copy link to this message
-
Re: Intermediate files generated.
Alex,

> I don't think this is what I am looking for. Essential, I wish to run both
> mapper as well as reducer. But at the same time, i wish to make sure that
> the temp files that are used between mappers and reducers are of my choice.
> Here, the choice means that I can specify the files in HDFS that can be used
> as temp files.

Could you explain why you want to do this ?

>
> thanks,
> --PB.
>
> On Fri, Jul 2, 2010 at 12:14 AM, Alex Loddengaard <[EMAIL PROTECTED]> wrote:
>
>> You could use the HDFS API from within your mapper, and run with 0
>> reducers.
>>
>> Alex
>>
>> On Thu, Jul 1, 2010 at 3:07 PM, Pramy Bhats <[EMAIL PROTECTED]
>> >wrote:
>>
>> > Hi,
>> >
>> > I am using hadoop framework for writing MapReduce jobs. I want  to
>> redirect
>> > the output of Map into files of my choice and later use those files as
>> > input
>> > for Reduce phase.
>> >
>> >
>> > Could you please suggest, how to proceed for it ?
>> >
>> > thanks,
>> > --PB.
>> >
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB