Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - simultaneous read + write?


Copy link to this message
-
Re: simultaneous read + write?
Scott Carey 2011-09-03, 00:58
AvroDataFile deals with this for some cases.  Is it an acceptable API for
your use case?  You can configure the block size to be very small and/or
flush() regularly.

If you do this on your own, you will need to track the position that you
start to read a record at, and if there is a failure, rewind and reset the
reader to that position.

-Scott
On 8/25/11 7:17 PM, "Yang" <[EMAIL PROTECTED]> wrote:

>I'm trying to implement an on-disk queue, which contains avro records,
>SpecificRecord
>
>my queue implementation basically contains a
>SpecificDatumWriter, and a SpecificDatumReader  pointing to the same file
>.
>
>the problem is, that when the reader reaches the EOF, I can no longer
>use it again,
>even after I append more records to the file,  if I call the same
>SpecificDatumReader.read() again,
>it gave me exceptions:
>
>
>--------------------------------------------------------------------------
>-----
>Test set: blah.MyTest
>--------------------------------------------------------------------------
>-----
>Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.257
>sec <<< FAILURE!
>testBasic(blah.MyTest)  Time elapsed: 0.24 sec  <<< ERROR!
>java.lang.ArrayIndexOutOfBoundsException
>    at java.lang.System.arraycopy(Native Method)
>    at
>org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill(BinaryDecoder.j
>ava:670)
>    at
>org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:453)
>    at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:120)
>    at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:405)
>    at
>org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
>    at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>    at
>org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
>    at
>org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:14
>2)
>    at
>org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.j
>ava:166)
>    at
>org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:13
>8)
>    at
>org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:12
>9)
>    at blah.DiskEventsQueue.dequeue2(MyTest.java:55)
>    at blah.MyTest.testBasic(MyTest.java:85)
>    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    at
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>57)
>    at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:43)
>    at java.lang.reflect.Method.invoke(Method.java:616)
>
>
>
>
>Thanks
>Yang