I was wondering if anyone has had any experience with porting cuda code to
hadoop pipes. Any assistance would be greatly appreciated.

Thanks,
--
View this message in context: http://old.nabble.com/HADOOP-PIPES-with-CUDA-tp33316352p33316352.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Do you mean porting existing cuda code away from Cuda to just some language
like python using pipes?  Or creating a solution that uses pipes to chain
mappers / reducers together, where the mappers and/or reducers invoke
Cuda kernels?  Or something else entirely?

You could do something like the second example, if you had a Cuda capable
card on each machine in the hadoop cluster.  And you might want to limit
you number of mappers / reducers running consecutively on each node to 1
per node, since the Cuda kernels execute serially (or at least the used
to...its been about 3 years since I've done any Cude coding).  That is
unless you had multiple graphics cards on each machine in the cluster.

Also, if you wanted to invoke the Cuda kernel in your reducer, you'd
probably need to spin through the entire Iterator of values and build up
your data structure in memory before you write it to the graphics card.
 This could possibly cause out of memory exceptions in your reducer if the
set of data sent to each reducer was too large to hold in memory.

Can you give an example of what you are interested in?

On Mon, Feb 13, 2012 at 9:02 AM, jem85 <[EMAIL PROTECTED]> wrote:

>
> I was wondering if anyone has had any experience with porting cuda code to
> hadoop pipes. Any assistance would be greatly appreciated.
>
> Thanks,
> --
> View this message in context:
> http://old.nabble.com/HADOOP-PIPES-with-CUDA-tp33316352p33316352.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
--

Thanks,
John C

On 02/13/2012 06:02 PM, jem85 wrote:
>
> I was wondering if anyone has had any experience with porting cuda code to
> hadoop pipes. Any assistance would be greatly appreciated.
>
> Thanks,

Probably not exactly what you're looking for, but we've ported other C
code to Hadoop.  We used Pydoop (http://pydoop.sourceforge.net/) for it,
since it runs C-Python over pipes and thus lets you use ctypes to easily
wrap C code.  It could be an option.  One could even consider using
PyCUDA with Pydoop.

Disclaimer:  I'm one of the Pydoop developers.
--
Luca Pireddu
CRS4 - Distributed Computing Group
Loc. Pixina Manna Edificio 1
09010 Pula (CA), Italy
Tel: +39 0709250452