Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> footer info in avro


Copy link to this message
-
Re: footer info in avro
There's no plan I know of to add this.

Avro's original file format wrote metadata at the end of the file.
This was changed in Avro 1.3 so that files could always be processed
sequentially, without seeking to the end.

Doug

On Wed, Oct 9, 2013 at 5:08 PM, Venkat <[EMAIL PROTECTED]> wrote:
> Hi All
>
> Does Avro file format support storing some json metadata at the end of the
> file (footer)? Currently, the header stores schema in json enabling
> consumers to augment it with application specific info. But I'm not sure if
> there is a way consumer can store specific metadata at the end of the file.
>
> The use case is that data processing applications can compute statistics
> during the scan and write the computed statistics at the end of the file
> before closing the stream. This avoid storing statistics else where.
>
> If this feature does not exits, is there a plan on adding it sometime in the
> future?
>
> Thanks a lot.