-footer info in avro
Venkat 2013-10-10, 00:08
Does Avro file format support storing some json metadata at the end of the file (footer)? Currently, the header stores schema in json enabling consumers to augment it with application specific info. But I'm not sure if there is a way consumer can store specific metadata at the end of the file.
The use case is that data processing applications can compute statistics during the scan and write the computed statistics at the end of the file before closing the stream. This avoid storing statistics else where.
If this feature does not exits, is there a plan on adding it sometime in the future?
Thanks a lot.