Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - Simple AvroStorage LOAD and STORE with Avro 1.6.0


+
Andrew Kenworthy 2012-01-09, 09:15
+
Stan Rosenberg 2012-01-09, 16:30
+
Andrew Kenworthy 2012-01-10, 13:03
Copy link to this message
-
Re: Simple AvroStorage LOAD and STORE with Avro 1.6.0
Stan Rosenberg 2012-01-10, 16:36
Andrew,

Something looks odd in this stack trace:

Caused by: java.lang.ClassCastException:
org.apache.pig.data.BinSedesTuple cannot be cast to
org.apache.avro.generic.IndexedRecord
>         at org.apache.avro.generic.GenericData.getField(GenericData.java:525)
>         at org.apache.avro.generic.GenericData.getField(GenericData.java:540)
>         at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:103)
>         at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
>         at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.write(PigAvroDatumWriter.java:99)

PigAvroDatumWriter overrides 'GenericDatumWriter.writeRecord' in order
to extract values from a tuple.  Thus, I would expect that the third
method invocation be PigAvroDatumWriter.writeRecord.  Perhaps, someone
else has more insight as to why it's not getting invoked.  In the
meantime, please confirm that both PigAvroDatumWriter and
GenericDatumWriter are loaded from the right jar files. (You can do
this by temporarily changing the pig script to invoke JVM with 'java
-verbose' and 'grep' the output for these classes.)

Best,

stan

On Tue, Jan 10, 2012 at 8:03 AM, Andrew Kenworthy
<[EMAIL PROTECTED]> wrote:
> Hi Stan,
>
> here's the full stacktrace:
>
> org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to org.apache.avro.generic.IndexedRecord
>         at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:261)
>         at org.apache.pig.piggybank.storage.avro.PigAvroRecordWriter.write(PigAvroRecordWriter.java:49)
>         at org.apache.pig.piggybank.storage.avro.AvroStorage.putNext(AvroStorage.java:580)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
>         at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:530)
>         at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
>         at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to org.apache.avro.generic.IndexedRecord
>         at org.apache.avro.generic.GenericData.getField(GenericData.java:525)
>         at org.apache.avro.generic.GenericData.getField(GenericData.java:540)
>         at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:103)
>         at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
>         at org.apache.pig.piggybank.storage.avro.PigAvroDatumWriter.write(PigAvroDatumWriter.java:99)
>         at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
>         at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:255)
+
Andrew Kenworthy 2012-01-11, 09:47
+
Scott Carey 2012-01-16, 21:16
+
Russell Jurney 2012-01-09, 20:47
+
Stan Rosenberg 2012-01-09, 20:52
+
Russell Jurney 2012-01-09, 21:21
+
Bill Graham 2012-01-10, 00:42