Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - One petabyte of data loading into HDFS with in 10 min.


Copy link to this message
-
Re: One petabyte of data loading into HDFS with in 10 min.
Fabio Pitzolu 2012-09-05, 14:47
290 days per petabyte, I'll analyze your data manually!! Also print out
some report! :-D
Fabio
2012/9/5 Cosmin Lehene <[EMAIL PROTECTED]>

> Here's an extremely naïve ballpark estimation: at theoretical hardware
> speed, for 3PB representing 1PB with 3x replication
>
> Over a single 1Gbps connection (and I'm not sure, you can actually reach
> 1Gbps)
> (3 petabytes) / (1 Gbps) = 291.271111 days
>
> So you'd need at least 40,000 1Gbps network cards to get that in 10
> minutes :) - (3PB/1Gbps)/40000<http://www.google.ro/search?client=safari&rls=en&q=(3PB/1Gbps)/40000&ie=UTF-8&oe=UTF-8&redir_esc=&ei=2WRHUNWtGIWo0QW52oDYDw>
>
> The actual number of nodes would depend a lot on the actual network
> architecture, the type of storage you use (SSD,  HDD), etc.
>
> Cosmin
> From: prabhu K <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Wednesday, September 5, 2012 3:21 PM
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: One petabyte of data loading into HDFS with in 10 min.
>
> Hi Users,
>
> Please clarify the below questions.
>
> 1. With in 10 minutes one petabyte of data load into HDFS/HIVE , how many
> slave (Data Nodes) machines required.
>
> 2. With in 10 minutes one petabyte of data load into HDFS/HIVE, what is
> the configuration setup for cloud computing.
>
> Please suggest and help me on this.
>
> Thanks&Regards,
> Prabhu.
>
>