I would say the hard limit is due to the OS local file system (and your
budget).

So short answer for ext3 : it doesn't seems so.
http://en.wikipedia.org/wiki/Ext3

And I am not sure the answer is the most interesting. Even if you could put
1 Peta on one node, what is usually interesting is the ratio
storage/compute.

Bertrand

On Wed, Jan 30, 2013 at 9:08 AM, jeba earnest <[EMAIL PROTECTED]> wrote:

>
> Hi,
>
>
> Is it possible to keep 1 Petabyte in a single data node?
> If not, How much is the maximum storage for a particular data node?
>
> Regards,
> M. Jeba
>

What would be the reason you would do that?

You would want to leverage distributed dataset for higher availability and better response times.

The maximum storage depends completely on the disks  capacity of your nodes and what your OS supports. Typically I have heard of about 1-2 TB/node to start with, but I may be wrong.
-abhishek
From: jeba earnest <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>, jeba earnest <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Wednesday, January 30, 2013 1:38 PM
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Maximum Storage size in a Single datanode
Hi,
Is it possible to keep 1 Petabyte in a single data node?
If not, How much is the maximum storage for a particular data node?

Regards,
M. Jeba

Hi Jeba,

 

There are other considerations too, for example, if a single node holds 1 PB
of data and if it were to die this would cause a significant amount of
traffic as NameNode arranges for new replicas to be created.

 

Vijay

 

From: Bertrand Dechoux [mailto:[EMAIL PROTECTED]]
Sent: 30 January 2013 09:14
To: [EMAIL PROTECTED]; jeba earnest
Subject: Re: Maximum Storage size in a Single datanode

 

I would say the hard limit is due to the OS local file system (and your
budget).

So short answer for ext3 : it doesn't seems so.
http://en.wikipedia.org/wiki/Ext3

And I am not sure the answer is the most interesting. Even if you could put
1 Peta on one node, what is usually interesting is the ratio
storage/compute.

Bertrand

On Wed, Jan 30, 2013 at 9:08 AM, jeba earnest <[EMAIL PROTECTED]> wrote:

 

Hi,

Is it possible to keep 1 Petabyte in a single data node?

If not, How much is the maximum storage for a particular data node?

 

Regards,
M. Jeba

 

I want to use either UBUNTU or REDHAT .
I just want to know how much storage space we can allocate in a single data node.

Is there any limitations in hadoop for storage in single node?

 
Regards,

Jeba
________________________________
 From: "Pamecha, Abhishek" <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; jeba earnest <[EMAIL PROTECTED]>
Sent: Wednesday, 30 January 2013 2:45 PM
Subject: Re: Maximum Storage size in a Single datanode
 

What would be the reason you would do that? 

You would want to leverage distributed dataset for higher availability and better response times.

The maximum storage depends completely on the disks  capacity of your nodes and what your OS supports. Typically I have heard of about 1-2 TB/node to start with, but I may be wrong.
-abhishek

From: jeba earnest <[EMAIL PROTECTED]>
Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, jeba earnest <[EMAIL PROTECTED]>
Date: Wednesday, January 30, 2013 1:38 PM
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Subject: Maximum Storage size in a Single datanode
Hi,

Is it possible to keep 1 Petabyte in a single data node?

If not, How much is the maximum storage for a particular data node?
 
Regards,
M. Jeba

You should probably think about this in a more cluster fashion.  A single
node with a PB of data is probably not a good allocation of CPU : Disk
ration.  In addition, you need enough RAM on your NameNode to keep track of
all of your blocks.  A few nodes with a PB each would quickly drive up NN
RAM requirements.

As others have mentioned, the local file system that HDFS sits on top of
may have limits.  We're going to use EXT4 which should handle that much,
but it's probably still not a good idea.

If you're just thinking of storing lots of data, you might consider
GlusterFS instead.

I highly recommend RedHat over Ubuntu.

Hope that helps.

On Wed, Jan 30, 2013 at 5:40 AM, jeba earnest <[EMAIL PROTECTED]> wrote:

>
> I want to use either UBUNTU or REDHAT .
> I just want to know how much storage space we can allocate in a single
> data node.
>
> Is there any limitations in hadoop for storage in single node?
>
>
>
> Regards,
> Jeba
>   ------------------------------
> *From:* "Pamecha, Abhishek" <[EMAIL PROTECTED]>
> *To:* "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; jeba earnest <
> [EMAIL PROTECTED]>
> *Sent:* Wednesday, 30 January 2013 2:45 PM
> *Subject:* Re: Maximum Storage size in a Single datanode
>
>  What would be the reason you would do that?
>
>  You would want to leverage distributed dataset for higher availability
> and better response times.
>
>  The maximum storage depends completely on the disks  capacity of your
> nodes and what your OS supports. Typically I have heard of about 1-2
> TB/node to start with, but I may be wrong.
> -abhishek
>
>
>   From: jeba earnest <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, jeba earnest
> <[EMAIL PROTECTED]>
> Date: Wednesday, January 30, 2013 1:38 PM
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Maximum Storage size in a Single datanode
>
>
>  Hi,
>
>
>  Is it possible to keep 1 Petabyte in a single data node?
>  If not, How much is the maximum storage for a particular data node?
>
> Regards,
> M. Jeba
>
>
>