Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Avro to Tuples during UnitTest


Copy link to this message
-
Re: Avro to Tuples during UnitTest
If you don't want to run the whole Pig script, then you probably shouldn't
use the AvroStorage, because it loads data to the Pig as per my
understanding. But as far as I understand you want to avoid the Pig
execution environment and just execute the UDF in isolation.
Probably you need to use the core Avro Java library, not the Pig+Avro
integration. In that case you would still need to write some conversion to
Pig tuples:(

Hope that helps,
Ruslan
On Fri, Aug 30, 2013 at 11:22 AM, Serega Sheypak
<[EMAIL PROTECTED]>wrote:

> 1. I want to test logic of my UDF without running the whole pig. I can set
> breakpoints, e.t.c.
> 2. UDF accepts a bag of tuples. I have source data which I want to read
> using AvroStorage to tuples and feed to UDF. Avro storage would preserve
> schema. I don;t have to wrie cusotm converters/cast'ers e,t,c,
>
> >>Or you can extract the
> >>logic to a java method outside of the UDF and test it within normal junit
> Who will convert sample avro data for me to Tuples and feed them to java
> method? I don't want to invent the wheel and repeat AvroStorage
> functionality.
>
>
> 2013/8/30 Ruslan Al-Fakikh <[EMAIL PROTECTED]>
>
> > Hi,
> >
> > What exactly do you want to test? The logic inside UDFs? In that case I
> > would recommend not bothering about input format of the whole Pig script.
> > You can use plain text files as input for the test. Or you can extract
> the
> > logic to a java method outside of the UDF and test it within normal junit
> > tests. Also these ideas come to my mind:
> > 1) You can take a look at PigUnit unility
> > 2) Avro has a human-readable (json text, non-binary) form for debugging
> > purposes
> >
> > Best Regards,
> > Ruslan
> >
> >
> > On Thu, Aug 29, 2013 at 2:56 PM, Serega Sheypak <
> [EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi, we have itegration test java utility which helps to test pig
> scripts.
> > > We often develop different UDFs in java and I would like to create unit
> > > tests for them. Right now they are tested with pig scripts during
> > > integration tests.
> > >
> > > I have a pack of prepared avro files with etalon input for my java
> UDFs.
> > > I would like to use some utility which would help me to convert avro to
> > > tuple list.
> > > I'm trying to use org.apache.pig.piggybank.storage.avro.AvroStorage
> with
> > no
> > > luck. I don't understand how it works.
> > >
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB