Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # user - Avro container format WAL with the Java API


Copy link to this message
-
Avro container format WAL with the Java API
Mike Percy 2012-10-09, 02:33
Hi all,
I am trying to implement a very simple WAL using the Avro container format.
Since my records are small, at startup time I seek backward by 256 bytes
from the end of the file before calling DataFileReader.sync(), and then run
through the available records to find the last one. I am writing a sync
marker and flushing after every record in with my writer. I was just
wondering, does this seem like a reasonable approach? Or is there a better
way to simply read the last record from the file?

Code snippet:

  private final DataFileReader<TransferStateFileMeta> reader;
  // ...

  /** Read the last record in the file */
  private void initReader() throws IOException {
    long syncPos = metaFile.length() - 256L;
    if (syncPos < 0) syncPos = 0L;
    reader.sync(syncPos);
    while (reader.hasNext()) {
      reader.next(metaCache);
    }
  }

Regards,
Mike