Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # dev >> Doubt regarding use of HTTP during Shuffle phase


+
Pavan Kulkarni 2012-06-27, 16:30
+
Pavan Kulkarni 2012-06-27, 16:33
+
Owen OMalley 2012-06-27, 16:40
+
Pavan Kulkarni 2012-06-27, 16:44
Copy link to this message
-
Re: Doubt regarding use of HTTP during Shuffle phase
On Wed, Jun 27, 2012 at 9:44 AM, Pavan Kulkarni <[EMAIL PROTECTED]> wrote:
> Yes you are correct, but we can use Lustre FS and it does scale right?
> I am new to this so please excuse if I am wrong in some assumptions.

You can use hybrid approaches, but you'd be venturing into the unknown.
Take a look at these projects first:
    http://code.google.com/p/sailfish/
    http://www.xyratex.com/pdfs/whitepapers/Xyratex_white_paper_MapReduce_1-4.pdf

Thanks,
Roman.
+
Pavan Kulkarni 2012-06-27, 16:57
+
Owen OMalley 2012-06-27, 17:21
+
Pavan Kulkarni 2012-06-27, 17:30
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB