Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Simple AvroStorage LOAD and STORE with Avro 1.6.0


+
Andrew Kenworthy 2012-01-09, 09:15
+
Stan Rosenberg 2012-01-09, 16:30
+
Andrew Kenworthy 2012-01-10, 13:03
+
Stan Rosenberg 2012-01-10, 16:36
+
Andrew Kenworthy 2012-01-11, 09:47
+
Scott Carey 2012-01-16, 21:16
+
Russell Jurney 2012-01-09, 20:47
Copy link to this message
-
Re: Simple AvroStorage LOAD and STORE with Avro 1.6.0
Generally, AvroStorage works fine for us with Avro 1.6.  However, we
also patched AvroStorage on a couple of occasions, e.g., see PIG-2330.

stan

On Mon, Jan 9, 2012 at 3:47 PM, Russell Jurney <[EMAIL PROTECTED]> wrote:
> I could only make AvroStorage work with Avro 1.4.1.
>
> Russell Jurney
> twitter.com/rjurney
> [EMAIL PROTECTED]
> datasyndrome.com
>
> On Jan 9, 2012, at 1:16 AM, Andrew Kenworthy <[EMAIL PROTECTED]> wrote:
>
> Hallo,
>
> When I run a simple pig script to LOAD and STORE avro data, I get:-
>
> java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be
> cast to org.apache.avro.generic.IndexedRecord
>
> Script:
>
> REGISTER /tmp/avro-1.6.0.jar;
> --REGISTER  /tmp/avro-1.5.4.jar
> --REGISTER  /tmp/avro-1.4.1.jar;
>
> REGISTER /tmp/piggybank-0.9.1.jar;
> REGISTER /tmp/json-simple-1.1.jar;
> REGISTER /tmp/jackson-core-asl-1.8.4.jar;
> REGISTER /tmp/jackson-mapper-asl-1.8.4.jar;
>
> avroData=LOAD '$DATA_INPUTDIR' USING
> org.apache.pig.piggybank.storage.avro.AvroStorage();
>
> dataSubset = FOREACH avroData GENERATE myField1, myField2;
> describe  dataSubset;
> -----------------------------------------------
> -- shows:
> -- dataSubset : { myField1: int, myField2: int}
> -----------------------------------------------
> STORE dataSubset INTO '$OUTPUTDIR' USING
> org.apache.pig.piggybank.storage.avro.AvroStorage();
>
> If I use the 1.5.4 jar I get the same error, but the script works with the
> 1.4.1 version. If I just write one field, then it works with 1.6.0.
>
> I see there's been a related issue fixed here:
>
> https://issues.apache.org/jira/browse/PIG-2202
> https://issues.apache.org/jira/browse/PIG-2195
>
> Can anyone confirm that this or similar works with avro 1.6.0, and/or point
> me in the right direction concering where the problem may lie?
>
> Many thanks,
>
> Andrew
+
Russell Jurney 2012-01-09, 21:21
+
Bill Graham 2012-01-10, 00:42