Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Simple AvroStorage LOAD and STORE with Avro 1.6.0


+
Andrew Kenworthy 2012-01-09, 09:15
+
Stan Rosenberg 2012-01-09, 16:30
+
Andrew Kenworthy 2012-01-10, 13:03
+
Stan Rosenberg 2012-01-10, 16:36
+
Andrew Kenworthy 2012-01-11, 09:47
+
Scott Carey 2012-01-16, 21:16
+
Russell Jurney 2012-01-09, 20:47
Copy link to this message
-
Re: Simple AvroStorage LOAD and STORE with Avro 1.6.0
Generally, AvroStorage works fine for us with Avro 1.6.  However, we
also patched AvroStorage on a couple of occasions, e.g., see PIG-2330.

stan

On Mon, Jan 9, 2012 at 3:47 PM, Russell Jurney <[EMAIL PROTECTED]> wrote:
> I could only make AvroStorage work with Avro 1.4.1.
>
> Russell Jurney
> twitter.com/rjurney
> [EMAIL PROTECTED]
> datasyndrome.com
>
> On Jan 9, 2012, at 1:16 AM, Andrew Kenworthy <[EMAIL PROTECTED]> wrote:
>
> Hallo,
>
> When I run a simple pig script to LOAD and STORE avro data, I get:-
>
> java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be
> cast to org.apache.avro.generic.IndexedRecord
>
> Script:
>
> REGISTER /tmp/avro-1.6.0.jar;
> --REGISTER  /tmp/avro-1.5.4.jar
> --REGISTER  /tmp/avro-1.4.1.jar;
>
> REGISTER /tmp/piggybank-0.9.1.jar;
> REGISTER /tmp/json-simple-1.1.jar;
> REGISTER /tmp/jackson-core-asl-1.8.4.jar;
> REGISTER /tmp/jackson-mapper-asl-1.8.4.jar;
>
> avroData=LOAD '$DATA_INPUTDIR' USING
> org.apache.pig.piggybank.storage.avro.AvroStorage();
>
> dataSubset = FOREACH avroData GENERATE myField1, myField2;
> describe  dataSubset;
> -----------------------------------------------
> -- shows:
> -- dataSubset : { myField1: int, myField2: int}
> -----------------------------------------------
> STORE dataSubset INTO '$OUTPUTDIR' USING
> org.apache.pig.piggybank.storage.avro.AvroStorage();
>
> If I use the 1.5.4 jar I get the same error, but the script works with the
> 1.4.1 version. If I just write one field, then it works with 1.6.0.
>
> I see there's been a related issue fixed here:
>
> https://issues.apache.org/jira/browse/PIG-2202
> https://issues.apache.org/jira/browse/PIG-2195
>
> Can anyone confirm that this or similar works with avro 1.6.0, and/or point
> me in the right direction concering where the problem may lie?
>
> Many thanks,
>
> Andrew
+
Russell Jurney 2012-01-09, 21:21
+
Bill Graham 2012-01-10, 00:42
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB