Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # dev - Mysterious 7 byte reads from .meta files


+
Varun Sharma 2013-07-06, 00:25
Copy link to this message
-
Re: Mysterious 7 byte reads from .meta files
Harsh J 2013-07-07, 16:06
The header of the meta file is read for block metadata version
validation. The checksum checks are performed after the header is read
(and skipped if the config's so). The code can be viewed in the
BlockReaderLocal.java, which is used when local reads are turned on.

I suppose we could skip reading the version as well, though it seems
"general". Please file a JIRA for this!

On Sat, Jul 6, 2013 at 5:55 AM, Varun Sharma <[EMAIL PROTECTED]> wrote:
> Hi,
>
> We are running hbase with application level checksums turned on. We
> basically have the following configuration:
>
> dfs.client.read.shortcircuit.skip.checksum -> true
> hbase.regionserver.checksum.verify -> true
>
> We have shortcircuit reads enabled and have verified that its working -
> only the HBase region server is accessing the disk. However, an strace on
> the region server shows some mysterious 7 byte seeks on the .meta files
> (which are also supposed to contain the checksum). We are running
> hadoop-2.0.0-alpha.
>
> Do 7 byte reads from .meta files ring a bell to someone's ears - I am not
> totally sure if these are checksums or not and I also don't know enough
> about hdfs ?
>
> Thanks
> Varun

--
Harsh J
+
Varun Sharma 2013-07-07, 16:37