I haven’t done profiling. The major overhead I can see is the conversion from ColumnVectorBatch to Arrow’s RecordBatch, which involves memory copy and some transcoding. Also the current adapter only supports reading entire stripe as a batch, which in a lot of cases is not ideal. I agree that we should maintain backward compatibility. I am thinking if we could expose another set of interface for Arrow which is built on top of the same ColumnReader/ColumnWriter classes.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB