Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - Avro 1.6.1 BinaryEncoder.writeBytes doesn't work on OS X


Copy link to this message
-
Re: Avro 1.6.1 BinaryEncoder.writeBytes doesn't work on OS X
Scott Carey 2012-02-08, 17:50
Yes, it appears as though GenericDatumWriter does not support native
ByteBuffers.  FileChannel returns native ByteBuffers, not HeapByteBuffer.

Some options:
* Read the file contents as a byte[], and wrap a ByteBuffer around that
* Create a new heap ByteBuffer and copy the contents of the file channel's
byte buffer into it.

Please file a bug in JIRA,

Thanks!

On 2/8/12 6:52 AM, "karthik ramachandran" <[EMAIL PROTECTED]> wrote:

> Hi all,
>
> I'm running into a problem with BinaryEncoder.writeBytes.  I'm trying to wrap
> some documents in an Avro structure and then write them into an Avro contain
> file.
>
> My code looks roughly like this :
>
>         File file = new File(...);
>         ClassLoader cl = SimpleAvroExample.class.
> getClassLoader();
>         InputStream isSimpleWrapperSchema> cl.getResourceAsStream("simple_wrapper.avpr");
>        
>         Schema.Parser parser = new Schema.Parser();
>         Schema schemaSimpleWrapper = parser.parse(isSimpleWrapperSchema);
>         GenericDatumWriter<GenericRecord> gdw = new
> GenericDatumWriter<GenericRecord>(schemaSimpleWrapper);
>         DataFileWriter writer = new DataFileWriter(gdw);
>
>         File outputFile = new File(...);
>         outputFile.createNewFile();
>        
>         writer.create(schemaSimpleWrapper, outputFile);
>        
>        
>         for(File documentFile : file.listFiles())
>         {
>        
>             GenericRecord r =  new GenericData.Record(schemaSimpleWrapper);
>             FileInputStream fis = new FileInputStream(documentFile);
>             FileChannel channel = fis.getChannel();
>            
>             r.put("filename", documentFile.getName());
>        
>              //Copying the whole file.
>             ByteBuffer byteBuffer = channel.map(FileChannel.MapMode.READ_ONLY,
> 0, documentFile.length());
>             r.put("body", byteBuffer);
>            
>            
>          
>            
>             writer.append(r);
>      
>             channel.close();
>             fis.close();
>            
>         }
>
> This code exceptions out :
>       Exception in thread "main"
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.UnsupportedOperationException
>         at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:261)
>         at org.iqt.cdl.SimpleAvroExample.main(SimpleAvroExample.java:107)
>      Caused by: java.lang.UnsupportedOperationException
>         at java.nio.ByteBuffer.arrayOffset(ByteBuffer.java:968)
>        at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:61)
>        at
> org.apache.avro.generic.GenericDatumWriter.writeBytes(GenericDatumWriter.java:
> 199)
>        at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:76)
>        at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java
> :105)
>        at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
>        at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
>        at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:255)
>        ... 1 more
>
>
> The problem seems to be that the writeBytes function on line 61 does the
> following :
>      int start = bytes.arrayOffset() + pos.
>
> Where byes is a java.nio.ByteBuffer.
>
> According to the JavaDocs for ByteBuffer the arrayOffset fuctional is
> optional.  
>
> As near as I can tell this OS X implementation of ByteBuffer does not support
> his functionality.
>
>
> --
> Karthik Ramachandran
> Mobile: 412-606-8981