Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # user - Is it possible to append to an already existing avro file


+
Vyacheslav Zholudev 2012-02-20, 21:45
+
Harsh J 2012-02-21, 04:29
Copy link to this message
-
Re: Is it possible to append to an already existing avro file
Vyacheslav Zholudev 2012-02-21, 15:29
Yep, I saw that method as well as the stackoverflow post. However, I'm interested how to append to a file on the arbitrary file system, not only on the local one.

I want to get an OutputStream based on the Path and the FileSystem implementation and then pass it for appending to avro methods.

Is that possible?

Thanks,
Vyacheslav

On Feb 21, 2012, at 5:29 AM, Harsh J wrote:

> Hi,
>
> Use the appendTo feature of the DataFileWriter. See
> http://avro.apache.org/docs/1.6.2/api/java/org/apache/avro/file/DataFileWriter.html#appendTo(java.io.File)
>
> For a quick setup example, read also:
> http://stackoverflow.com/questions/8806689/can-you-append-data-to-an-existing-avro-data-file
>
> On Tue, Feb 21, 2012 at 3:15 AM, Vyacheslav Zholudev
> <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>> is it possible to append to an already existing avro file when it was
>> written and closed before?
>>
>> If I use
>> outputStream = fs.append(avroFilePath);
>>
>> then later on I get: java.io.IOException: Invalid sync!
>>
>> Probably because the schema is written twice and some other issues.
>>
>> If I use outputStream = fs.create(avroFilePath); then the avro file gets
>> overwritten.
>>
>> Thanks,
>> Vyacheslav
>
>
>
> --
> Harsh J
> Customer Ops. Engineer
> Cloudera | http://tiny.cloudera.com/about
+
Scott Carey 2012-02-21, 17:02
+
Vyacheslav Zholudev 2012-02-22, 09:57
+
Michael Malak 2013-02-01, 19:32