Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> simultaneous read + write?

Copy link to this message
Re: simultaneous read + write?
AvroDataFile deals with this for some cases.  Is it an acceptable API for
your use case?  You can configure the block size to be very small and/or
flush() regularly.

If you do this on your own, you will need to track the position that you
start to read a record at, and if there is a failure, rewind and reset the
reader to that position.

On 8/25/11 7:17 PM, "Yang" <[EMAIL PROTECTED]> wrote:

>I'm trying to implement an on-disk queue, which contains avro records,
>my queue implementation basically contains a
>SpecificDatumWriter, and a SpecificDatumReader  pointing to the same file
>the problem is, that when the reader reaches the EOF, I can no longer
>use it again,
>even after I append more records to the file,  if I call the same
>SpecificDatumReader.read() again,
>it gave me exceptions:
>Test set: blah.MyTest
>Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.257
>sec <<< FAILURE!
>testBasic(blah.MyTest)  Time elapsed: 0.24 sec  <<< ERROR!
>    at java.lang.System.arraycopy(Native Method)
>    at
>    at
>    at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:120)
>    at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:405)
>    at
>    at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>    at
>    at
>    at
>    at
>    at
>    at blah.DiskEventsQueue.dequeue2(MyTest.java:55)
>    at blah.MyTest.testBasic(MyTest.java:85)
>    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    at
>    at
>    at java.lang.reflect.Method.invoke(Method.java:616)