Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Avro 1.6.1 BinaryEncoder.writeBytes doesn't work on OS X


Copy link to this message
-
Re: Avro 1.6.1 BinaryEncoder.writeBytes doesn't work on OS X
Yes, it appears as though GenericDatumWriter does not support native
ByteBuffers.  FileChannel returns native ByteBuffers, not HeapByteBuffer.

Some options:
* Read the file contents as a byte[], and wrap a ByteBuffer around that
* Create a new heap ByteBuffer and copy the contents of the file channel's
byte buffer into it.

Please file a bug in JIRA,

Thanks!

On 2/8/12 6:52 AM, "karthik ramachandran" <[EMAIL PROTECTED]> wrote:

> Hi all,
>
> I'm running into a problem with BinaryEncoder.writeBytes.  I'm trying to wrap
> some documents in an Avro structure and then write them into an Avro contain
> file.
>
> My code looks roughly like this :
>
>         File file = new File(...);
>         ClassLoader cl = SimpleAvroExample.class.
> getClassLoader();
>         InputStream isSimpleWrapperSchema> cl.getResourceAsStream("simple_wrapper.avpr");
>        
>         Schema.Parser parser = new Schema.Parser();
>         Schema schemaSimpleWrapper = parser.parse(isSimpleWrapperSchema);
>         GenericDatumWriter<GenericRecord> gdw = new
> GenericDatumWriter<GenericRecord>(schemaSimpleWrapper);
>         DataFileWriter writer = new DataFileWriter(gdw);
>
>         File outputFile = new File(...);
>         outputFile.createNewFile();
>        
>         writer.create(schemaSimpleWrapper, outputFile);
>        
>        
>         for(File documentFile : file.listFiles())
>         {
>        
>             GenericRecord r =  new GenericData.Record(schemaSimpleWrapper);
>             FileInputStream fis = new FileInputStream(documentFile);
>             FileChannel channel = fis.getChannel();
>            
>             r.put("filename", documentFile.getName());
>        
>              //Copying the whole file.
>             ByteBuffer byteBuffer = channel.map(FileChannel.MapMode.READ_ONLY,
> 0, documentFile.length());
>             r.put("body", byteBuffer);
>            
>            
>          
>            
>             writer.append(r);
>      
>             channel.close();
>             fis.close();
>            
>         }
>
> This code exceptions out :
>       Exception in thread "main"
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.UnsupportedOperationException
>         at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:261)
>         at org.iqt.cdl.SimpleAvroExample.main(SimpleAvroExample.java:107)
>      Caused by: java.lang.UnsupportedOperationException
>         at java.nio.ByteBuffer.arrayOffset(ByteBuffer.java:968)
>        at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:61)
>        at
> org.apache.avro.generic.GenericDatumWriter.writeBytes(GenericDatumWriter.java:
> 199)
>        at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:76)
>        at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java
> :105)
>        at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
>        at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
>        at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:255)
>        ... 1 more
>
>
> The problem seems to be that the writeBytes function on line 61 does the
> following :
>      int start = bytes.arrayOffset() + pos.
>
> Where byes is a java.nio.ByteBuffer.
>
> According to the JavaDocs for ByteBuffer the arrayOffset fuctional is
> optional.  
>
> As near as I can tell this OS X implementation of ByteBuffer does not support
> his functionality.
>
>
> --
> Karthik Ramachandran
> Mobile: 412-606-8981
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB