Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: Hadoop noob question


+
Rahul Bhattacharjee 2013-05-11, 16:10
Copy link to this message
-
Hadoop noob question
Hi All,

Can anyone help me know how does companies like Facebook ,Yahoo etc upload
bulk files say to the tune of 100 petabytes to Hadoop HDFS cluster for
processing
and after processing how they download those files from HDFS to local file
system.

I don't think they might be using the command line hadoop fs put to upload
files as it would take too long or do they divide say 10 parts each 10
petabytes and  compress and use the command line hadoop fs put

Or if they use any tool to upload huge files.

Please help me .

Thanks
thoihen
+
Nitin Pawar 2013-05-11, 10:54
+
maisnam ns 2013-05-11, 11:08
+
Nitin Pawar 2013-05-11, 11:24
+
Mohammad Tariq 2013-05-12, 13:42
+
Rahul Bhattacharjee 2013-05-12, 11:53
+
Nitin Pawar 2013-05-12, 12:06
+
Mohammad Tariq 2013-05-12, 12:37
+
Rahul Bhattacharjee 2013-05-12, 12:45
+
Mohammad Tariq 2013-05-12, 12:55
+
Chris Mawata 2013-05-12, 14:21
+
Rahul Bhattacharjee 2013-05-16, 14:18
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB