Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Send Data and Message to all nodes


Copy link to this message
-
Re: Send Data and Message to all nodes
Michael Segel 2012-01-29, 14:17
Unless I'm missing something, it sounds like the OP wants to chain jobs where the results from one job are the input to another...

Of course it's Sun morning and I haven't had my first cup of coffee so I could be misinterpreting the OP's question.

If the OP wanted to send the data to each node and use it as a lookup table, his initial output is on HDFS so he could just open the file and read it in to memory in Mapper.setup().  
Note: if the file is too big, then you probably wouldn't want to use distributed cache any way...
Sent from my iPhone

On Jan 28, 2012, at 7:11 PM, "Ravi Prakash" <[EMAIL PROTECTED]> wrote:

> Take a look at distributed cache for distributing data to all nodes. I'm
> not sure what you mean by messages. The MR programming paradigm is
> different from MPI.
>
> http://hadoop.apache.org/common/docs/r0.18.3/mapred_tutorial.html#DistributedCache
>
> On Sat, Jan 28, 2012 at 5:52 AM, Oliaei <[EMAIL PROTECTED]> wrote:
>
>>
>> Hi,
>>
>> I want to run a MR procedure under Hadoop and then send some messages &
>> data
>> to all of nodes and after that run anther MR.
>> What's the easiest way for sending data to all or some nodes? Or "Is there
>> any way to do that under Hadoop without using other frameworks?"
>>
>> Regards,
>>
>> Oliaei
>>
>> [EMAIL PROTECTED]
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Send-Data-and-Message-to-all-nodes-tp33219535p33219535.html
>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>
>>