Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> HDFS block storage


+
John Lilley 2013-07-15, 18:23
Copy link to this message
-
Re: HDFS block storage
1) Right now, I would say jira and code.
2) It is not really a flat storage. It 'folds' itself when needed.
3) At least checksums. There are jiras about whether it should be somehow
in the block itself.

A beginning : https://issues.apache.org/jira/browse/HADOOP-1134
A (dead?) discussion : https://issues.apache.org/jira/browse/HDFS-2699

Regards

Bertrand

On Mon, Jul 15, 2013 at 8:23 PM, John Lilley <[EMAIL PROTECTED]>wrote:

>  I was looking at the HDFS block storage and noticed a couple things (1)
> all block files are in a flat directory structure (2) there is a “meta”
> file for each block file.  This leads me to ask:****
>
> -- Where can I find good reading that describes this level of HDFS
> internals?****
>
> -- Is the flat storage sufficient to handle millions of blocks?****
>
> -- What is in the meta file?  Why isn’t it just a header or trailer on the
> data file?****
>
> Thanks****
>
> John****
>
> ** **
>

--
Bertrand Dechoux
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB