Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> footer info in avro


Copy link to this message
-
footer info in avro
Hi All

Does Avro file format support storing some json metadata at the end of the file (footer)? Currently, the header stores schema in json enabling consumers to augment it with application specific info. But I'm not sure if there is a way consumer can store specific metadata at the end of the file.

The use case is that data processing applications can compute statistics during the scan and write the computed statistics at the end of the file before closing the stream. This avoid storing statistics else where. 

If this feature does not exits, is there a plan on adding it sometime in the future?

Thanks a lot. 
+
Doug Cutting 2013-10-10, 00:29