Hello,

I'm looking at HDFS as a potential canditate to store files shared by few hundreds of users.
This idea would be to mount local drive directly linked to the HDFS cloud. The datanodes would be deployed on each desktops which means up to 500 DNs.

Now that said browsing the web it seems that :
- HDFS is design to store large files (block at a minimum of 64Mb)
>- HDFS has a SPOC with the NN
>- HDFS security seems to be very limited
In such a situation, the average file size would be 500KB. So i would think of using a 1MB block.

Does anybody have experience running such a configuration ? Feedback ?
Any opinion ? Should i compltely drop that idea ?
Would the NN accept 500 mounts through FUSE ?

Thx & brgds

Hi Franck,

This is not HDFS's intended use case and I do not think it will solve
your problem very successfully.

You're probably better off looking at some other technologies.

-Todd

On Thu, Nov 10, 2011 at 4:52 AM, Franck Besnard <[EMAIL PROTECTED]> wrote:
> Hello,
> I'm looking at HDFS as a potential canditate to store files shared by few
> hundreds of users.
> This idea would be to mount local drive directly linked to the HDFS cloud.
> The datanodes would be deployed on each desktops which means up to 500 DNs.
> Now that said browsing the web it seems that :
>
> - HDFS is design to store large files (block at a minimum of 64Mb)
> - HDFS has a SPOC with the NN
> - HDFS security seems to be very limited
>
> In such a situation, the average file size would be 500KB. So i would think
> of using a 1MB block.
> Does anybody have experience running such a configuration ? Feedback ?
> Any opinion ? Should i compltely drop that idea ?
> Would the NN accept 500 mounts through FUSE ?
> Thx & brgds
>

--
Todd Lipcon
Software Engineer, Cloudera

As Todd said, HDFS isn't suited to this. You could take a look at Gluster though. It seems like it would fit your needs better.

-Ivan