Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: One petabyte of data loading into HDFS with in 10 min.


+
Nick Jones 2012-09-05, 14:59
+
Mathias Herberts 2012-09-05, 15:12
+
zGreenfelder 2012-09-05, 14:56
+
DSouza, Clive V 2012-09-05, 14:58
+
Michael Segel 2012-09-07, 14:00
+
prabhu K 2012-09-10, 07:40
+
Steve Loughran 2012-09-10, 09:40
+
Michael Segel 2012-09-10, 11:50
+
Gauthier, Alexander 2012-09-10, 16:17
Copy link to this message
-
Re: One petabyte of data loading into HDFS with in 10 min.
Fabio Pitzolu 2012-09-05, 14:47
290 days per petabyte, I'll analyze your data manually!! Also print out
some report! :-D
Fabio
2012/9/5 Cosmin Lehene <[EMAIL PROTECTED]>

> Here's an extremely naïve ballpark estimation: at theoretical hardware
> speed, for 3PB representing 1PB with 3x replication
>
> Over a single 1Gbps connection (and I'm not sure, you can actually reach
> 1Gbps)
> (3 petabytes) / (1 Gbps) = 291.271111 days
>
> So you'd need at least 40,000 1Gbps network cards to get that in 10
> minutes :) - (3PB/1Gbps)/40000<http://www.google.ro/search?client=safari&rls=en&q=(3PB/1Gbps)/40000&ie=UTF-8&oe=UTF-8&redir_esc=&ei=2WRHUNWtGIWo0QW52oDYDw>
>
> The actual number of nodes would depend a lot on the actual network
> architecture, the type of storage you use (SSD,  HDD), etc.
>
> Cosmin
> From: prabhu K <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Wednesday, September 5, 2012 3:21 PM
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: One petabyte of data loading into HDFS with in 10 min.
>
> Hi Users,
>
> Please clarify the below questions.
>
> 1. With in 10 minutes one petabyte of data load into HDFS/HIVE , how many
> slave (Data Nodes) machines required.
>
> 2. With in 10 minutes one petabyte of data load into HDFS/HIVE, what is
> the configuration setup for cloud computing.
>
> Please suggest and help me on this.
>
> Thanks&Regards,
> Prabhu.
>
>
+
prabhu K 2012-09-05, 12:21
+
Chen He 2012-09-05, 14:03
+
Shailesh Dargude 2012-09-05, 14:14
+
Mohammad Tariq 2012-09-05, 14:22
+
Steve Loughran 2012-09-07, 09:12
+
Gulfie 2012-09-06, 20:52
+
Michael Segel 2012-09-10, 19:54