Hi,
 I am working on a project where I am trying to create hardlinks for the
Reducer nodes to the intermediate output files after Map phase during the
Shuffle phase
instead of  copying the files via HTTP which I assume would be more
efficient.
 I had few doubts regarding this. Why is HTTP used during the Shuffle phase
instead of just creating a Hardlink?
and Can you give pointers towards how do I create these hardlinks? I am
looking for where exactly can the intermediate Mapoutput Filename be
fetched?
Any help is highly appreciated .Thanks
--

--With Regards
Pavan Kulkarni

Hi,
 I am working on a project where I am trying to create hardlinks for the
Reducer nodes to the intermediate output files after Map phase during the
Shuffle phase
instead of  copying the files via HTTP which I assume would be more
efficient.
 I had few doubts regarding this. Why is HTTP used during the Shuffle phase
instead of just creating a Hardlink?
and Can you give pointers towards how do I create these hardlinks? I am
looking for where exactly can the intermediate Mapoutput Filename be
fetched?
Any help is highly appreciated .Thanks

--

--With Regards
Pavan Kulkarni

On Wed, Jun 27, 2012 at 9:33 AM, Pavan Kulkarni <[EMAIL PROTECTED]>wrote:

Why is HTTP used during the Shuffle phase
> instead of just creating a Hardlink?
>

The reduces run on a separate machine that the map. Hadoop doesn't use a
shared NFS file system between the machines, because it doesn't scale.

-- Owen

Yes you are correct, but we can use Lustre FS and it does scale right?
I am new to this so please excuse if I am wrong in some assumptions.Thanks

On Wed, Jun 27, 2012 at 9:40 AM, Owen O'Malley <[EMAIL PROTECTED]> wrote:

> On Wed, Jun 27, 2012 at 9:33 AM, Pavan Kulkarni <[EMAIL PROTECTED]
> >wrote:
>
> Why is HTTP used during the Shuffle phase
> > instead of just creating a Hardlink?
> >
>
> The reduces run on a separate machine that the map. Hadoop doesn't use a
> shared NFS file system between the machines, because it doesn't scale.
>
> -- Owen
>

--

--With Regards
Pavan Kulkarni

On Wed, Jun 27, 2012 at 9:44 AM, Pavan Kulkarni <[EMAIL PROTECTED]> wrote:
> Yes you are correct, but we can use Lustre FS and it does scale right?
> I am new to this so please excuse if I am wrong in some assumptions.

You can use hybrid approaches, but you'd be venturing into the unknown.
Take a look at these projects first:
    http://code.google.com/p/sailfish/
    http://www.xyratex.com/pdfs/whitepapers/Xyratex_white_paper_MapReduce_1-4.pdf

Thanks,
Roman.

@Roman: Exactly these are the papers I referred before venturing into this
project.
Thanks a lot for the sailfish paper.Wasn't aware of that.
Also by any chance do you have any idea what classes do I need to tweak to
get the Filenames of Map Output files.?Thanks

On Wed, Jun 27, 2012 at 9:52 AM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:

> On Wed, Jun 27, 2012 at 9:44 AM, Pavan Kulkarni <[EMAIL PROTECTED]>
> wrote:
> > Yes you are correct, but we can use Lustre FS and it does scale right?
> > I am new to this so please excuse if I am wrong in some assumptions.
>
> You can use hybrid approaches, but you'd be venturing into the unknown.
> Take a look at these projects first:
>    http://code.google.com/p/sailfish/
>
> http://www.xyratex.com/pdfs/whitepapers/Xyratex_white_paper_MapReduce_1-4.pdf
>
> Thanks,
> Roman.
>

--

--With Regards
Pavan Kulkarni

Pavan,
  This is a very big project. Look at the users of IFile.java. IFile is the
format for storing the shuffle outputs.

-- Owen

Oh.Thanks a lot Owen.
I'll have a look into it.

On Wed, Jun 27, 2012 at 10:21 AM, Owen O'Malley <[EMAIL PROTECTED]> wrote:

> Pavan,
>  This is a very big project. Look at the users of IFile.java. IFile is the
> format for storing the shuffle outputs.
>
> -- Owen
>

--

--With Regards
Pavan Kulkarni