Hello,

I noticed that the shuffle phase is reading data over http even when data
is available locally. The version of hadoop I'm using is 1.0.3. Is there a
reason it is implemented this way ? Is it OK to make a change that will
identify that the data is available locally and read from the local disk
instead of the http?

I'm new to this developer list and apache developer list in general. So
please feel free to let me know if there is a certain etiquette that I'm
not following.

Thanks,
Suresh.

It's just using the loopback address, right -not going on to the external
network and back again?

On 22 January 2013 03:22, Suresh Kumar <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I noticed that the shuffle phase is reading data over http even when data
> is available locally. The version of hadoop I'm using is 1.0.3. Is there a
> reason it is implemented this way ? Is it OK to make a change that will
> identify that the data is available locally and read from the local disk
> instead of the http?
>
> I'm new to this developer list and apache developer list in general. So
> please feel free to let me know if there is a certain etiquette that I'm
> not following.
>
> Thanks,
> Suresh.
>

Hi Steve,

My assumption is that unless it is reading from  http://127.0.0.1/ or
http://localhost/ , it reads over the network. If I'm wrong please correct
me. The http tracker address that a ReduceTask receives is not of that
format. So I do not think it is reading using the loop back address.

Thanks,
Suresh.

On Tue, Jan 22, 2013 at 8:46 AM, Steve Loughran <[EMAIL PROTECTED]>wrote:

> It's just using the loopback address, right -not going on to the external
> network and back again?
>
> On 22 January 2013 03:22, Suresh Kumar <[EMAIL PROTECTED]> wrote:
>
> > Hello,
> >
> > I noticed that the shuffle phase is reading data over http even when data
> > is available locally. The version of hadoop I'm using is 1.0.3. Is there
> a
> > reason it is implemented this way ? Is it OK to make a change that will
> > identify that the data is available locally and read from the local disk
> > instead of the http?
> >
> > I'm new to this developer list and apache developer list in general. So
> > please feel free to let me know if there is a certain etiquette that I'm
> > not following.
> >
> > Thanks,
> > Suresh.
> >
>

I have a patch that tries to use file links instead of making a copy of the
data that is already available locally. I tested it on the a single machine
cluster configuration running 48 mappers and reducers. I unfortunately do
not have access to a cluster even a small one. Can some on review and test
run my patch ?

I created the patch using Eclipse against 1.0.3. My knowledge in Java in
limited and the code is not well written in some classes. So please let me
know if I need to make changes to the code along with a short explanation
of the change.  I will happily do so.

Thanks,
Suresh.

You can setup the right /etc/hosts to support the loopback. OTOH, saving
disk io would be more important for small clusters with large instances.
Hadoop historically works on large clusters with relatively small
instances, so the issue was not as acute. MAPREDUCE-4049 allows the shuffle
to be pluggable, so you won't have to patch Hadoop framework code itself.

Are you saying that you don't have access to EC2?
On Tue, Jan 22, 2013 at 11:02 AM, Suresh Kumar <[EMAIL PROTECTED]>wrote:

> I have a patch that tries to use file links instead of making a copy of
> the data that is already available locally. I tested it on the a single
> machine cluster configuration running 48 mappers and reducers. I
> unfortunately do not have access to a cluster even a small one. Can some on
> review and test run my patch ?
>
> I created the patch using Eclipse against 1.0.3. My knowledge in Java in
> limited and the code is not well written in some classes. So please let me
> know if I need to make changes to the code along with a short explanation
> of the change.  I will happily do so.
>
> Thanks,
> Suresh.
>
>
>
>

I've experimented with similar changes in the hadoop trunk, although my
desire was to improve performance for networked file systems.  I had not
considered the idea that it could be used for files stored locally on
disk.

What type of performance tests did you run and what kind of improvements
did you find (or not find)?

Al

On Tue, 2013-01-22 at 11:02 -0800, Suresh Kumar wrote:
> I have a patch that tries to use file links instead of making a copy
> of the data that is already available locally. I tested it on the a
> single machine cluster configuration running 48 mappers and reducers.
> I unfortunately do not have access to a cluster even a small one. Can
> some on review and test run my patch ?
>
>
> I created the patch using Eclipse against 1.0.3. My knowledge in Java
> in limited and the code is not well written in some classes. So please
> let me know if I need to make changes to the code along with a short
> explanation of the change.  I will happily do so.
>
>
> Thanks,
> Suresh.
>
>
>
>
>
--
Albert Chu
[EMAIL PROTECTED]
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory

Hi Luke,

I checked the /etc/hosts and it is configured correctly. Looks like the
slow shuffle read speeds we were getting are due to slow disk IO.

I will go through the change MAPREDUCE-4049 and see if I can update my
patch to work with that code on version 3.0.0

I did not think of EC2, that is a good idea.

Thanks,
Suresh.

On Tue, Jan 22, 2013 at 11:24 AM, Luke Lu <[EMAIL PROTECTED]> wrote:

> You can setup the right /etc/hosts to support the loopback. OTOH, saving
> disk io would be more important for small clusters with large instances.
> Hadoop historically works on large clusters with relatively small
> instances, so the issue was not as acute. MAPREDUCE-4049 allows the shuffle
> to be pluggable, so you won't have to patch Hadoop framework code itself.
>
> Are you saying that you don't have access to EC2?
>
>
> On Tue, Jan 22, 2013 at 11:02 AM, Suresh Kumar <[EMAIL PROTECTED]
> >wrote:
>
> > I have a patch that tries to use file links instead of making a copy of
> > the data that is already available locally. I tested it on the a single
> > machine cluster configuration running 48 mappers and reducers. I
> > unfortunately do not have access to a cluster even a small one. Can some
> on
> > review and test run my patch ?
> >
> > I created the patch using Eclipse against 1.0.3. My knowledge in Java in
> > limited and the code is not well written in some classes. So please let
> me
> > know if I need to make changes to the code along with a short explanation
> > of the change.  I will happily do so.
> >
> > Thanks,
> > Suresh.
> >
> >
> >
> >
>

Hi Al,

As Luke mentioned the change I made is very useful for small clusters with
lots of cores. I'm working on very ideal case ie 1 machine cluster with 48
cores So I really do not know how it would be most general use cases.

In one of my use cases, the shuffle copy used to take between 40 mins. It
now takes 10-30 seconds. In another use case with map almost close to an
identity function the unpatched the shuffle copy lasted for 12+hours before
it failed as it ran out of disk space. Using the patched code the shuffle
copy lasted about 30 -60 seconds.

Thanks,
Suresh.

On Tue, Jan 22, 2013 at 11:42 AM, Albert Chu <[EMAIL PROTECTED]> wrote:

> I've experimented with similar changes in the hadoop trunk, although my
> desire was to improve performance for networked file systems.  I had not
> considered the idea that it could be used for files stored locally on
> disk.
>
> What type of performance tests did you run and what kind of improvements
> did you find (or not find)?
>
> Al
>
> On Tue, 2013-01-22 at 11:02 -0800, Suresh Kumar wrote:
> > I have a patch that tries to use file links instead of making a copy
> > of the data that is already available locally. I tested it on the a
> > single machine cluster configuration running 48 mappers and reducers.
> > I unfortunately do not have access to a cluster even a small one. Can
> > some on review and test run my patch ?
> >
> >
> > I created the patch using Eclipse against 1.0.3. My knowledge in Java
> > in limited and the code is not well written in some classes. So please
> > let me know if I need to make changes to the code along with a short
> > explanation of the change.  I will happily do so.
> >
> >
> > Thanks,
> > Suresh.
> >
> >
> >
> >
> >
> --
> Albert Chu
> [EMAIL PROTECTED]
> Computer Scientist
> High Performance Systems Division
> Lawrence Livermore National Laboratory
>
>