Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # dev - Shuffling over the network for local map data.


Copy link to this message
-
Re: Shuffling over the network for local map data.
Suresh Kumar 2013-01-22, 17:36
Hi Steve,

My assumption is that unless it is reading from  http://127.0.0.1/ or
http://localhost/ , it reads over the network. If I'm wrong please correct
me. The http tracker address that a ReduceTask receives is not of that
format. So I do not think it is reading using the loop back address.

Thanks,
Suresh.

On Tue, Jan 22, 2013 at 8:46 AM, Steve Loughran <[EMAIL PROTECTED]>wrote:

> It's just using the loopback address, right -not going on to the external
> network and back again?
>
> On 22 January 2013 03:22, Suresh Kumar <[EMAIL PROTECTED]> wrote:
>
> > Hello,
> >
> > I noticed that the shuffle phase is reading data over http even when data
> > is available locally. The version of hadoop I'm using is 1.0.3. Is there
> a
> > reason it is implemented this way ? Is it OK to make a change that will
> > identify that the data is available locally and read from the local disk
> > instead of the http?
> >
> > I'm new to this developer list and apache developer list in general. So
> > please feel free to let me know if there is a certain etiquette that I'm
> > not following.
> >
> > Thanks,
> > Suresh.
> >
>