Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Doubt regarding use of HTTP during Shuffle phase


Copy link to this message
-
Re: Doubt regarding use of HTTP during Shuffle phase
On Wed, Jun 27, 2012 at 9:44 AM, Pavan Kulkarni <[EMAIL PROTECTED]> wrote:
> Yes you are correct, but we can use Lustre FS and it does scale right?
> I am new to this so please excuse if I am wrong in some assumptions.

You can use hybrid approaches, but you'd be venturing into the unknown.
Take a look at these projects first:
    http://code.google.com/p/sailfish/
    http://www.xyratex.com/pdfs/whitepapers/Xyratex_white_paper_MapReduce_1-4.pdf

Thanks,
Roman.