Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - Sane max storage size for DN


+
Mohammad Tariq 2012-12-12, 15:02
+
Ted Dunning 2012-12-12, 15:44
Copy link to this message
-
Re: Sane max storage size for DN
Mohammad Tariq 2012-12-12, 15:52
Thank you so much for the valuable response Ted.

No, there would be dedicated storage for NN as well.

Any tips on RAM & N/W?

*Computations are not really frequent.

Thanks again.

Regards,
    Mohammad Tariq

On Wed, Dec 12, 2012 at 9:14 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

>
> Yes it does make sense, depending on how much compute each byte of data
> will require on average.  With ordinary Hadoop, it is reasonable to have
> half a dozen 2TB drives.  With specialized versions of Hadoop considerably
> more can be supported.
>
> From what you say, it sounds like you are suggesting that your name node
> get a part of a single drive with the rest being shared with other virtual
> instances or with an OS partition.  That would be a really bad idea for
> performance.  Many Hadoop programs are I/O bound so having more than one
> spindle is a good thing.
>
>
>
> On Wed, Dec 12, 2012 at 7:02 AM, Mohammad Tariq <[EMAIL PROTECTED]>wrote:
>
>> Hello list,
>>
>>           I don't know if this question makes any sense, but I would like
>> to ask, does it make sense to store 500TB (or more) data in a single DN?If
>> yes, then what should be the spec of other parameters *viz*. NN & DN
>> RAM, N/W etc?If no, what could be the alternative?
>>
>> Many thanks.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>>
>
+
Michael Segel 2012-12-12, 18:58
+
Mohammad Tariq 2012-12-13, 03:52