Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - Avro to Tuples during UnitTest


+
Serega Sheypak 2013-08-29, 10:56
+
Ruslan Al-Fakikh 2013-08-30, 07:16
+
Serega Sheypak 2013-08-30, 07:22
+
Ruslan Al-Fakikh 2013-08-30, 07:34
Copy link to this message
-
Re: Avro to Tuples during UnitTest
Serega Sheypak 2013-08-30, 08:18
I understand you solution. I want to reuse exisitng code. This json with
schema to Tuple conversion utility would took to much time to implement.
2013/8/30 Ruslan Al-Fakikh <[EMAIL PROTECTED]>

> If you don't want to run the whole Pig script, then you probably shouldn't
> use the AvroStorage, because it loads data to the Pig as per my
> understanding. But as far as I understand you want to avoid the Pig
> execution environment and just execute the UDF in isolation.
> Probably you need to use the core Avro Java library, not the Pig+Avro
> integration. In that case you would still need to write some conversion to
> Pig tuples:(
>
> Hope that helps,
> Ruslan
>
>
> On Fri, Aug 30, 2013 at 11:22 AM, Serega Sheypak
> <[EMAIL PROTECTED]>wrote:
>
> > 1. I want to test logic of my UDF without running the whole pig. I can
> set
> > breakpoints, e.t.c.
> > 2. UDF accepts a bag of tuples. I have source data which I want to read
> > using AvroStorage to tuples and feed to UDF. Avro storage would preserve
> > schema. I don;t have to wrie cusotm converters/cast'ers e,t,c,
> >
> > >>Or you can extract the
> > >>logic to a java method outside of the UDF and test it within normal
> junit
> > Who will convert sample avro data for me to Tuples and feed them to java
> > method? I don't want to invent the wheel and repeat AvroStorage
> > functionality.
> >
> >
> > 2013/8/30 Ruslan Al-Fakikh <[EMAIL PROTECTED]>
> >
> > > Hi,
> > >
> > > What exactly do you want to test? The logic inside UDFs? In that case I
> > > would recommend not bothering about input format of the whole Pig
> script.
> > > You can use plain text files as input for the test. Or you can extract
> > the
> > > logic to a java method outside of the UDF and test it within normal
> junit
> > > tests. Also these ideas come to my mind:
> > > 1) You can take a look at PigUnit unility
> > > 2) Avro has a human-readable (json text, non-binary) form for debugging
> > > purposes
> > >
> > > Best Regards,
> > > Ruslan
> > >
> > >
> > > On Thu, Aug 29, 2013 at 2:56 PM, Serega Sheypak <
> > [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Hi, we have itegration test java utility which helps to test pig
> > scripts.
> > > > We often develop different UDFs in java and I would like to create
> unit
> > > > tests for them. Right now they are tested with pig scripts during
> > > > integration tests.
> > > >
> > > > I have a pack of prepared avro files with etalon input for my java
> > UDFs.
> > > > I would like to use some utility which would help me to convert avro
> to
> > > > tuple list.
> > > > I'm trying to use org.apache.pig.piggybank.storage.avro.AvroStorage
> > with
> > > no
> > > > luck. I don't understand how it works.
> > > >
> > >
> >
>