Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: One petabyte of data loading into HDFS with in 10 min.


+
Nick Jones 2012-09-05, 14:59
+
Mathias Herberts 2012-09-05, 15:12
+
zGreenfelder 2012-09-05, 14:56
+
DSouza, Clive V 2012-09-05, 14:58
+
Michael Segel 2012-09-07, 14:00
+
prabhu K 2012-09-10, 07:40
+
Steve Loughran 2012-09-10, 09:40
+
Michael Segel 2012-09-10, 11:50
+
Gauthier, Alexander 2012-09-10, 16:17
Copy link to this message
-
Re: One petabyte of data loading into HDFS with in 10 min.
290 days per petabyte, I'll analyze your data manually!! Also print out
some report! :-D
Fabio
2012/9/5 Cosmin Lehene <[EMAIL PROTECTED]>

> Here's an extremely naïve ballpark estimation: at theoretical hardware
> speed, for 3PB representing 1PB with 3x replication
>
> Over a single 1Gbps connection (and I'm not sure, you can actually reach
> 1Gbps)
> (3 petabytes) / (1 Gbps) = 291.271111 days
>
> So you'd need at least 40,000 1Gbps network cards to get that in 10
> minutes :) - (3PB/1Gbps)/40000<http://www.google.ro/search?client=safari&rls=en&q=(3PB/1Gbps)/40000&ie=UTF-8&oe=UTF-8&redir_esc=&ei=2WRHUNWtGIWo0QW52oDYDw>
>
> The actual number of nodes would depend a lot on the actual network
> architecture, the type of storage you use (SSD,  HDD), etc.
>
> Cosmin
> From: prabhu K <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Wednesday, September 5, 2012 3:21 PM
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: One petabyte of data loading into HDFS with in 10 min.
>
> Hi Users,
>
> Please clarify the below questions.
>
> 1. With in 10 minutes one petabyte of data load into HDFS/HIVE , how many
> slave (Data Nodes) machines required.
>
> 2. With in 10 minutes one petabyte of data load into HDFS/HIVE, what is
> the configuration setup for cloud computing.
>
> Please suggest and help me on this.
>
> Thanks&Regards,
> Prabhu.
>
>
+
prabhu K 2012-09-05, 12:21
+
Chen He 2012-09-05, 14:03
+
Shailesh Dargude 2012-09-05, 14:14
+
Mohammad Tariq 2012-09-05, 14:22
+
Steve Loughran 2012-09-07, 09:12
+
Gulfie 2012-09-06, 20:52
+
Michael Segel 2012-09-10, 19:54
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB