Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Matrix multiplication in Hadoop

Copy link to this message
Re: Matrix multiplication in Hadoop
Edward Capriolo 2011-11-18, 19:46
A problem with matrix multiplication in hadoop is that hadoop is row
oriented for the most part. I have thought about this use case however and
you can theoretically turn a 2D matrix into a 1D matrix and then that fits
into the row oriented nature of hadoop. Also being that the typical mapper
can have fairly large chunks of memory like 1024MB I have done work like
this before were I loaded such datasets into memory to process them. That
usage does not really fit the map reduce model.

I have been wanting to look at:

On Fri, Nov 18, 2011 at 1:48 PM, Ayon Sinha <[EMAIL PROTECTED]> wrote:

> I'd really be interested in a comparison of Numpy/Octave/Matlab kind of
> tools with a Hadoop (lets say 4-10 large cloud servers) implementation with
> growing size of the matrix. I want to know the scale at which Hadoop really
> starts to pull away.
> -Ayon
> See My Photos on Flickr
> Also check out my Blog for answers to commonly asked questions.
> ________________________________
> From: Michel Segel <[EMAIL PROTECTED]>
> Sent: Friday, November 18, 2011 9:33 AM
> Subject: Re: Matrix multiplication in Hadoop
> Is Hadoop the best tool for doing large matrix math.
> Sure you can do it, but, aren't there better tools for these types of
> problems?
> Sent from a remote device. Please excuse any typos...
> Mike Segel
> On Nov 18, 2011, at 10:59 AM, Mike Spreitzer <[EMAIL PROTECTED]> wrote:
> > Who is doing multiplication of large dense matrices using Hadoop?  What
> is
> > a good way to do that computation using Hadoop?
> >
> > Thanks,
> > Mike