Avro files have special byte-string markers that can be used to find the
beginning of the next block but it will add some complexity.  My
understanding is the Python Avro libraries are very slow.  You may want to
try a prototype in Java and see if that meets your performance needs.

Alternatively, you can try writing several small files and process those in
parallel.

Joshua

On Thu, Jul 5, 2018 at 5:11 PM Troy X <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB