Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Send Data and Message to all nodes


Copy link to this message
-
Re: Send Data and Message to all nodes
Unless I'm missing something, it sounds like the OP wants to chain jobs where the results from one job are the input to another...

Of course it's Sun morning and I haven't had my first cup of coffee so I could be misinterpreting the OP's question.

If the OP wanted to send the data to each node and use it as a lookup table, his initial output is on HDFS so he could just open the file and read it in to memory in Mapper.setup().  
Note: if the file is too big, then you probably wouldn't want to use distributed cache any way...
Sent from my iPhone

On Jan 28, 2012, at 7:11 PM, "Ravi Prakash" <[EMAIL PROTECTED]> wrote:

> Take a look at distributed cache for distributing data to all nodes. I'm
> not sure what you mean by messages. The MR programming paradigm is
> different from MPI.
>
> http://hadoop.apache.org/common/docs/r0.18.3/mapred_tutorial.html#DistributedCache
>
> On Sat, Jan 28, 2012 at 5:52 AM, Oliaei <[EMAIL PROTECTED]> wrote:
>
>>
>> Hi,
>>
>> I want to run a MR procedure under Hadoop and then send some messages &
>> data
>> to all of nodes and after that run anther MR.
>> What's the easiest way for sending data to all or some nodes? Or "Is there
>> any way to do that under Hadoop without using other frameworks?"
>>
>> Regards,
>>
>> Oliaei
>>
>> [EMAIL PROTECTED]
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Send-Data-and-Message-to-all-nodes-tp33219535p33219535.html
>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB