I was looking at the HDFS block storage and noticed a couple things (1) all block files are in a flat directory structure (2) there is a "meta" file for each block file. This leads me to ask:
-- Where can I find good reading that describes this level of HDFS internals?
-- Is the flat storage sufficient to handle millions of blocks?
-- What is in the meta file? Why isn't it just a header or trailer on the data file?