Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Sane max storage size for DN


+
Mohammad Tariq 2012-12-12, 15:02
Copy link to this message
-
Re: Sane max storage size for DN
Yes it does make sense, depending on how much compute each byte of data
will require on average.  With ordinary Hadoop, it is reasonable to have
half a dozen 2TB drives.  With specialized versions of Hadoop considerably
more can be supported.

>From what you say, it sounds like you are suggesting that your name node
get a part of a single drive with the rest being shared with other virtual
instances or with an OS partition.  That would be a really bad idea for
performance.  Many Hadoop programs are I/O bound so having more than one
spindle is a good thing.

On Wed, Dec 12, 2012 at 7:02 AM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> Hello list,
>
>           I don't know if this question makes any sense, but I would like
> to ask, does it make sense to store 500TB (or more) data in a single DN?If
> yes, then what should be the spec of other parameters *viz*. NN & DN RAM,
> N/W etc?If no, what could be the alternative?
>
> Many thanks.
>
> Regards,
>     Mohammad Tariq
>
>
>
+
Mohammad Tariq 2012-12-12, 15:52
+
Michael Segel 2012-12-12, 18:58
+
Mohammad Tariq 2012-12-13, 03:52
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB