Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Avro container format WAL with the Java API

Copy link to this message
Avro container format WAL with the Java API
Hi all,
I am trying to implement a very simple WAL using the Avro container format.
Since my records are small, at startup time I seek backward by 256 bytes
from the end of the file before calling DataFileReader.sync(), and then run
through the available records to find the last one. I am writing a sync
marker and flushing after every record in with my writer. I was just
wondering, does this seem like a reasonable approach? Or is there a better
way to simply read the last record from the file?

Code snippet:

  private final DataFileReader<TransferStateFileMeta> reader;
  // ...

  /** Read the last record in the file */
  private void initReader() throws IOException {
    long syncPos = metaFile.length() - 256L;
    if (syncPos < 0) syncPos = 0L;
    while (reader.hasNext()) {