Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - FSDataInputStream.read returns -1 with growing file and never continues reading


Copy link to this message
-
FSDataInputStream.read returns -1 with growing file and never continues reading
Christoph Rupp 2012-12-20, 12:21
Hi,

I am experiencing an unexpected situation where FSDataInputStream.read()
returns -1 while reading data from a file that another process still
appends to. According to the documentation read() should never return -1
but throw Exceptions on errors. In addition, there's more data available,
and read() definitely should not fail.

The problem gets worse because the FSDataInputStream is not able to recover
from this. If it once returns -1 then it will always return -1, even if the
file continues growing.

If, at the same time, other Java processes read other HDFS files, they will
also return -1 immediately after opening the file. It smells like this
error gets propagated to other client processes as well.

I found a workaround: close the FSDataInputStream, open it again and then
seek to the previous position. And then reading works fine.

Another problem that i have seen is that the FSDataInputStream returns -1
when reaching EOF. It will never return 0 (which i would expect when
reaching EOF).

I use CDH 4.1.2, but also saw this with CDH 3u5. I have attached samples to
reproduce this.

My cluster consists of 4 machines; 1 namenode and 3 datanodes. I run my
tests on the namenode machine. there are no other HDFS users, and the load
that is generated by my tests is fairly low, i would say.

One process writes to 6 files simultaneously, but with a 5 sec sleep
between each write. It uses an FSDataOutputStream, and after writing data
it calls sync(). Each write() appends 8 mb; it stops when the file grows to
100 mb.

Six processes read files; each process reads one file. At first each reader
loops till the file exists. If it does then it opens the FSDataInputStream
and starts reading. Usually the first process returns the first 8 MB in the
file before it starts returning -1. But the other processes immediately
return -1 without reading any data. I start the 6 reader processes before i
start the writer.

Search HdfsReader.java for "WORKAROUND" and remove the comments; this will
reopen the FSDataInputStream after -1 is returned, and then everything
works.

Sources are attached.

This is a very basic scenario and i wonder if i'm doing anything wrong or
if i found an HDFS bug.

bye
Christoph