-Re: [ANN] Plasma MapReduce, PlasmaFS, version 0.4
Steve Loughran 2011-10-13, 12:48
On 12/10/11 17:31, Gerd Stolpmann wrote:
> This is about the release of Plasma-0.4, an alternate and independent
> implementation of map/reduce with its own dfs. This might also be
> interesting for Hadoop users and developers, because this project
> incorporates a number of new ideas. So far, Plasma has proven to work on
> smaller clusters and shows good signs of being scalable. The design of
> PlasmaFS is certainly superior to that of HDFS - I did not want a
> quick'n'dirty solution, so please have a look how to do it right.
> Concerning the features, these two pages compare Plasma and Hadoop:
- without block checksums your code contains assumptions about HDD
integrity that does not stand up to the classic works by Pinhero or
Schroeder. Essentially you appear to be assuming that HDDs don't corrupt
data, yet both HDD and their interconnects can play up. For a recent
summary of Hadoop integrity, I would point you at [Loughran2011]
-Hadoop NNs benefit from SSD too.
-auth and security has improved recently, though I'd still run it in a
private subnet just to be sure
> I hope you see where the point is.
Again, support for small block size is relevant in small situations. In
larger clusters you will not only have larger block sizes, if you do
work on small blocks the sheer number of task trackers reporting back to
the JT can overload it.
> I have currently only limited resources for testing my implementation.
> If there is anybody interested in testing on bigger clusters, please let
> me know.
That's one of the issues with the Plasma design: I'm not sure how well
things like Posix semantics, esp. locking and writes with offsets scale.
That's why the very large filesystems, HDFS included, tend to drop them.
Look at how much effort it took to get Append to work reliably.
Without evidence of working at scale, I'm not sure how the claim "the
design of Plasma is certainly superior to HDFS" is defensible. Sorry.
That said, using SunOS RPC/NFS as an FS protocol is nice as it does make
mounting straightforward. And as NFS locking isn't guaranteed in NFS,
you may be able to get away without it.