Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - how to execute different tasks on data nodes(simultaneously in hadoop).


Copy link to this message
-
Re: how to execute different tasks on data nodes(simultaneously in hadoop).
Bertrand Dechoux 2012-09-03, 16:31
You can check the value of "map.input.file" in order to apply a different
logic for each type of files (in the mapper).
More information about your problem/context would help the readers to
provide a more extensive reply.

Regards

Bertrand

On Mon, Sep 3, 2012 at 6:25 PM, Michael Segel <[EMAIL PROTECTED]>wrote:

> Not sure what you are trying to do...
>
> You want to pass through the entire data set on all nodes where each node
> runs a single filter?
>
> You're thinking is orthogonal to how Hadoop works.
>
> You would be better off letting each node work on a portion of the data
> which is local to that node running the entire filter set.
>
>
> On Sep 3, 2012, at 11:19 AM, mallik arjun <[EMAIL PROTECTED]> wrote:
>
> > genrally in hadoop map function will be exeucted by all the data nodes
> on the input data set ,against this how can i do the following.
> > i have some filter programs , and what i want to do is each data
> node(slave) has to execute one filter alogrithm  simultaneously, diffent
> from other data nodes executions.
> >
> > thanks in advance.
> >
> >
>
>
--
Bertrand Dechoux