Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Avro container format WAL with the Java API


Copy link to this message
-
Avro container format WAL with the Java API
Hi all,
I am trying to implement a very simple WAL using the Avro container format.
Since my records are small, at startup time I seek backward by 256 bytes
from the end of the file before calling DataFileReader.sync(), and then run
through the available records to find the last one. I am writing a sync
marker and flushing after every record in with my writer. I was just
wondering, does this seem like a reasonable approach? Or is there a better
way to simply read the last record from the file?

Code snippet:

  private final DataFileReader<TransferStateFileMeta> reader;
  // ...

  /** Read the last record in the file */
  private void initReader() throws IOException {
    long syncPos = metaFile.length() - 256L;
    if (syncPos < 0) syncPos = 0L;
    reader.sync(syncPos);
    while (reader.hasNext()) {
      reader.next(metaCache);
    }
  }

Regards,
Mike
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB