Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Avro to Tuples during UnitTest


Copy link to this message
-
Re: Avro to Tuples during UnitTest
I understand you solution. I want to reuse exisitng code. This json with
schema to Tuple conversion utility would took to much time to implement.
2013/8/30 Ruslan Al-Fakikh <[EMAIL PROTECTED]>

> If you don't want to run the whole Pig script, then you probably shouldn't
> use the AvroStorage, because it loads data to the Pig as per my
> understanding. But as far as I understand you want to avoid the Pig
> execution environment and just execute the UDF in isolation.
> Probably you need to use the core Avro Java library, not the Pig+Avro
> integration. In that case you would still need to write some conversion to
> Pig tuples:(
>
> Hope that helps,
> Ruslan
>
>
> On Fri, Aug 30, 2013 at 11:22 AM, Serega Sheypak
> <[EMAIL PROTECTED]>wrote:
>
> > 1. I want to test logic of my UDF without running the whole pig. I can
> set
> > breakpoints, e.t.c.
> > 2. UDF accepts a bag of tuples. I have source data which I want to read
> > using AvroStorage to tuples and feed to UDF. Avro storage would preserve
> > schema. I don;t have to wrie cusotm converters/cast'ers e,t,c,
> >
> > >>Or you can extract the
> > >>logic to a java method outside of the UDF and test it within normal
> junit
> > Who will convert sample avro data for me to Tuples and feed them to java
> > method? I don't want to invent the wheel and repeat AvroStorage
> > functionality.
> >
> >
> > 2013/8/30 Ruslan Al-Fakikh <[EMAIL PROTECTED]>
> >
> > > Hi,
> > >
> > > What exactly do you want to test? The logic inside UDFs? In that case I
> > > would recommend not bothering about input format of the whole Pig
> script.
> > > You can use plain text files as input for the test. Or you can extract
> > the
> > > logic to a java method outside of the UDF and test it within normal
> junit
> > > tests. Also these ideas come to my mind:
> > > 1) You can take a look at PigUnit unility
> > > 2) Avro has a human-readable (json text, non-binary) form for debugging
> > > purposes
> > >
> > > Best Regards,
> > > Ruslan
> > >
> > >
> > > On Thu, Aug 29, 2013 at 2:56 PM, Serega Sheypak <
> > [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Hi, we have itegration test java utility which helps to test pig
> > scripts.
> > > > We often develop different UDFs in java and I would like to create
> unit
> > > > tests for them. Right now they are tested with pig scripts during
> > > > integration tests.
> > > >
> > > > I have a pack of prepared avro files with etalon input for my java
> > UDFs.
> > > > I would like to use some utility which would help me to convert avro
> to
> > > > tuple list.
> > > > I'm trying to use org.apache.pig.piggybank.storage.avro.AvroStorage
> > with
> > > no
> > > > luck. I don't understand how it works.
> > > >
> > >
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB