Hello,

How much memory/JVM heap does NameNode use for each block?

I've tried locating this in the FAQ and on search-hadoop.com, but couldn't find a ton of concrete numbers, just these two:

http://search-hadoop.com/m/RmxWMVyVvK1 - 150 bytes/block?
http://search-hadoop.com/m/O886P1VyVvK1 - 1 GB heap for every object?

Thanks,
Otis

> How much memory/JVM heap does NameNode use for each block?

I don't remember the exact number, it also depends on which version of
Hadoop you're using

> http://search-hadoop.com/m/O886P1VyVvK1 - 1 GB heap for every object?

It's 1 GB for every *million* objects (files, blocks, etc.). This is a
good rule of thumb, at least for the 0.20.x/1.0.0 series.

Is there a reason you need a more exact estimate?

-Joey

--
Joseph Echeverria
Cloudera, Inc.
443.305.9434

On Tue, Jan 17, 2012 at 10:08 AM, Otis Gospodnetic <
[EMAIL PROTECTED]> wrote:

> Hello,
>
> How much memory/JVM heap does NameNode use for each block?
>
> I've tried locating this in the FAQ and on search-hadoop.com, but
> couldn't find a ton of concrete numbers, just these two:
>
> http://search-hadoop.com/m/RmxWMVyVvK1 - 150 bytes/block?
> http://search-hadoop.com/m/O886P1VyVvK1 - 1 GB heap for every object?
>
> Thanks,
> Otis
>

Some real world statistics. From NN web Interface. replication factor=2

Cluster Summary
22,061,605 files and directories, 22,151,870 blocks = 44,213,475 total.
Heap Size is 10.85 GB / 16.58 GB (65%)

compressedOOps is enabled.

Konstantin's paper
http://www.usenix.org/publications/login/2010-04/openpdfs/shvachko.pdf

mentions that on average a file consumes about 600 bytes of memory in the
name-node (1 file object + 2 block objects).

To quote from his paper (see page 9)

".. in order to store 100 million files (referencing 200 million blocks) a
name-node should have at least 60GB of RAM. This matches observations on
deployed clusters".

On Tue, Jan 17, 2012 at 7:08 AM, Otis Gospodnetic <
[EMAIL PROTECTED]> wrote:

> Hello,
>
> How much memory/JVM heap does NameNode use for each block?
>
> I've tried locating this in the FAQ and on search-hadoop.com, but
> couldn't find a ton of concrete numbers, just these two:
>
> http://search-hadoop.com/m/RmxWMVyVvK1 - 150 bytes/block?
> http://search-hadoop.com/m/O886P1VyVvK1 - 1 GB heap for every object?
>
> Thanks,
> Otis
>

Does it mean that on an average 1 file has only 2 blocks ( with
replication=1 ) ?
On 1/18/12, M. C. Srivas <[EMAIL PROTECTED]> wrote:
> Konstantin's paper
> http://www.usenix.org/publications/login/2010-04/openpdfs/shvachko.pdf
>
> mentions that on average a file consumes about 600 bytes of memory in the
> name-node (1 file object + 2 block objects).
>
> To quote from his paper (see page 9)
>
> ".. in order to store 100 million files (referencing 200 million blocks) a
> name-node should have at least 60GB of RAM. This matches observations on
> deployed clusters".
>
>
>
> On Tue, Jan 17, 2012 at 7:08 AM, Otis Gospodnetic <
> [EMAIL PROTECTED]> wrote:
>
>> Hello,
>>
>> How much memory/JVM heap does NameNode use for each block?
>>
>> I've tried locating this in the FAQ and on search-hadoop.com, but
>> couldn't find a ton of concrete numbers, just these two:
>>
>> http://search-hadoop.com/m/RmxWMVyVvK1 - 150 bytes/block?
>> http://search-hadoop.com/m/O886P1VyVvK1 - 1 GB heap for every object?
>>
>> Thanks,
>> Otis
>>
>

--
Sent from my mobile device

how much blocks a file has depends on your data status,please consider it
according to your data status....
2012/1/18 prasenjit mukherjee <[EMAIL PROTECTED]>

> Does it mean that on an average 1 file has only 2 blocks ( with
> replication=1 ) ?
>
>
> On 1/18/12, M. C. Srivas <[EMAIL PROTECTED]> wrote:
> > Konstantin's paper
> > http://www.usenix.org/publications/login/2010-04/openpdfs/shvachko.pdf
> >
> > mentions that on average a file consumes about 600 bytes of memory in the
> > name-node (1 file object + 2 block objects).
> >
> > To quote from his paper (see page 9)
> >
> > ".. in order to store 100 million files (referencing 200 million blocks)
> a
> > name-node should have at least 60GB of RAM. This matches observations on
> > deployed clusters".
> >
> >
> >
> > On Tue, Jan 17, 2012 at 7:08 AM, Otis Gospodnetic <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Hello,
> >>
> >> How much memory/JVM heap does NameNode use for each block?
> >>
> >> I've tried locating this in the FAQ and on search-hadoop.com, but
> >> couldn't find a ton of concrete numbers, just these two:
> >>
> >> http://search-hadoop.com/m/RmxWMVyVvK1 - 150 bytes/block?
> >> http://search-hadoop.com/m/O886P1VyVvK1 - 1 GB heap for every object?
> >>
> >> Thanks,
> >> Otis
> >>
> >
>
> --
> Sent from my mobile device
>