Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # dev >> Shuffling over the network for local map data.


+
Suresh Kumar 2013-01-22, 03:22
+
Steve Loughran 2013-01-22, 16:46
+
Suresh Kumar 2013-01-22, 17:36
+
Suresh Kumar 2013-01-22, 19:02
Copy link to this message
-
Re: Shuffling over the network for local map data.
I've experimented with similar changes in the hadoop trunk, although my
desire was to improve performance for networked file systems.  I had not
considered the idea that it could be used for files stored locally on
disk.

What type of performance tests did you run and what kind of improvements
did you find (or not find)?

Al

On Tue, 2013-01-22 at 11:02 -0800, Suresh Kumar wrote:
> I have a patch that tries to use file links instead of making a copy
> of the data that is already available locally. I tested it on the a
> single machine cluster configuration running 48 mappers and reducers.
> I unfortunately do not have access to a cluster even a small one. Can
> some on review and test run my patch ?
>
>
> I created the patch using Eclipse against 1.0.3. My knowledge in Java
> in limited and the code is not well written in some classes. So please
> let me know if I need to make changes to the code along with a short
> explanation of the change.  I will happily do so.
>
>
> Thanks,
> Suresh.
>
>
>
>
>
--
Albert Chu
[EMAIL PROTECTED]
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory
+
Suresh Kumar 2013-01-22, 23:03
+
Luke Lu 2013-01-22, 19:24
+
Suresh Kumar 2013-01-22, 22:22