Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - RE: Maximum Storage size in a Single datanode


+
Vijay Thakorlal 2013-01-30, 12:16
+
Jean-Marc Spaggiari 2013-01-30, 12:21
Copy link to this message
-
Re: Maximum Storage size in a Single datanode
Fatih Haltas 2013-01-30, 12:43
I think, he just wants to learn the approximate storage capacity of each
datanode that he should configure , 1 PB is just a made up amount of
storage, I guess. Probably, in my opinion, he already knows that even not
caring hadoop, this is too much for any server, but he made just a made up
mistake :)

Dont keep on at him that much :)

30 Ocak 2013 Çarşamba tarihinde Jean-Marc Spaggiari adlı kullanıcı şöyle
yazdı:

> Hi,
>
> Also, think about the memory you will need in your DataNode to serve
> all this data... I'm not sure there is any server which can take that
> today. You need a certain amount of memory per block in the DN. With
> all this data, you will have SOOOO many blocks...
>
> Regarding RH vs Ubuntu, I think Ubuntu is more an end user
> distribution than a server one. And I found RH a bit "not enought
> free". I have installed Debian on all my servers.
>
> JM
>
> 2013/1/30, Vijay Thakorlal <[EMAIL PROTECTED]>:
> > Jeba,
> >
> >
> >
> > I'm not aware of any hadoop limitations in this respect (others may be
> able
> > to comment on this); since blocks are just files on the OS, the datanode
> > will create subdirectories to store blocks to avoid problems with large
> > numbers of files in a single directory. So I would think the limitations
> > are
> > primarily around the type of file system you select, for ext3 it
> > theoretically supports up to 16TB (http://en.wikipedia.org/wiki/Ext3)
> and
> > for ext4 up to 1EB (http://en.wikipedia.org/wiki/Ext4). Although you're
> > probably already planning on deploying 64-bit servers, I believe for
> large
> > FS on ext4 you'd be better off with a 64-bit server.
> >
> >
> >
> > As far as OS is concerned anecdotally (based on blogs, hadoop mailing
> lists
> > etc) I believe there are more production deployments using RHEL and/or
> > CentOS than Ubuntu.
> >
> >
> >
> > It's probably not practical to have nodes with 1PB of data for the
> reasons
> > that others have mentioned and due to the replication traffic that will
> be
> > generated if the node dies. Not to mention fsck times with large file
> > systems.
> >
> >
> >
> > Vijay
> >
> >
> >
> >
> >
> >
> >
> > From: jeba earnest [mailto:[EMAIL PROTECTED]]
> > Sent: 30 January 2013 10:40
> > To: [EMAIL PROTECTED]
> > Subject: Re: Maximum Storage size in a Single datanode
> >
> >
> >
> >
> >
> > I want to use either UBUNTU or REDHAT .
> >
> > I just want to know how much storage space we can allocate in a single
> data
> > node.
> >
> >
> >
> > Is there any limitations in hadoop for storage in single node?
> >
> >
> >
> >
> >
> >
> >
> > Regards,
> >
> > Jeba
> >
> >   _____
> >
> > From: "Pamecha, Abhishek" <[EMAIL PROTECTED]>
> > To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; jeba earnest
> > <[EMAIL PROTECTED]>
> > Sent: Wednesday, 30 January 2013 2:45 PM
> > Subject: Re: Maximum Storage size in a Single datanode
> >
> >
> >
> > What would be the reason you would do that?
> >
> >
> >
> > You would want to leverage distributed dataset for higher availability
> and
> > better response times.
> >
> >
> >
> > The maximum storage depends completely on the disks  capacity of your
> nodes
> > and what your OS supports. Typically I have heard of about 1-2 TB/node to
> > start with, but I may be wrong.
> >
> > -abhishek
> >
> >
> >
> >
> >
> > From: jeba earnest <[EMAIL PROTECTED]>
> > Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>, jeba
> earnest
> > <[EMAIL PROTECTED]>
> > Date: Wednesday, January 30, 2013 1:38 PM
> > To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> > Subject: Maximum Storage size in a Single datanode
> >
> >
> >
> >
> >
> > Hi,
> >
> >
> >
> > Is it possible to keep 1 Petabyte in a single data node?
> >
> > If not, How much is the maximum storage for a particular data node?
> >
> >
> >
> > Regards,
> > M. Jeba
> >
> >
> >
> >
>
+
Mohammad Tariq 2013-01-30, 12:02