Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> simultaneous read + write?


Copy link to this message
-
Re: simultaneous read + write?
AvroDataFile deals with this for some cases.  Is it an acceptable API for
your use case?  You can configure the block size to be very small and/or
flush() regularly.

If you do this on your own, you will need to track the position that you
start to read a record at, and if there is a failure, rewind and reset the
reader to that position.

-Scott
On 8/25/11 7:17 PM, "Yang" <[EMAIL PROTECTED]> wrote:

>I'm trying to implement an on-disk queue, which contains avro records,
>SpecificRecord
>
>my queue implementation basically contains a
>SpecificDatumWriter, and a SpecificDatumReader  pointing to the same file
>.
>
>the problem is, that when the reader reaches the EOF, I can no longer
>use it again,
>even after I append more records to the file,  if I call the same
>SpecificDatumReader.read() again,
>it gave me exceptions:
>
>
>--------------------------------------------------------------------------
>-----
>Test set: blah.MyTest
>--------------------------------------------------------------------------
>-----
>Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.257
>sec <<< FAILURE!
>testBasic(blah.MyTest)  Time elapsed: 0.24 sec  <<< ERROR!
>java.lang.ArrayIndexOutOfBoundsException
>    at java.lang.System.arraycopy(Native Method)
>    at
>org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill(BinaryDecoder.j
>ava:670)
>    at
>org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:453)
>    at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:120)
>    at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:405)
>    at
>org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
>    at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>    at
>org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
>    at
>org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:14
>2)
>    at
>org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.j
>ava:166)
>    at
>org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:13
>8)
>    at
>org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:12
>9)
>    at blah.DiskEventsQueue.dequeue2(MyTest.java:55)
>    at blah.MyTest.testBasic(MyTest.java:85)
>    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    at
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>57)
>    at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:43)
>    at java.lang.reflect.Method.invoke(Method.java:616)
>
>
>
>
>Thanks
>Yang
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB