Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Avro to Tuples during UnitTest


+
Serega Sheypak 2013-08-29, 10:56
+
Ruslan Al-Fakikh 2013-08-30, 07:16
Copy link to this message
-
Re: Avro to Tuples during UnitTest
1. I want to test logic of my UDF without running the whole pig. I can set
breakpoints, e.t.c.
2. UDF accepts a bag of tuples. I have source data which I want to read
using AvroStorage to tuples and feed to UDF. Avro storage would preserve
schema. I don;t have to wrie cusotm converters/cast'ers e,t,c,

>>Or you can extract the
>>logic to a java method outside of the UDF and test it within normal junit
Who will convert sample avro data for me to Tuples and feed them to java
method? I don't want to invent the wheel and repeat AvroStorage
functionality.
2013/8/30 Ruslan Al-Fakikh <[EMAIL PROTECTED]>

> Hi,
>
> What exactly do you want to test? The logic inside UDFs? In that case I
> would recommend not bothering about input format of the whole Pig script.
> You can use plain text files as input for the test. Or you can extract the
> logic to a java method outside of the UDF and test it within normal junit
> tests. Also these ideas come to my mind:
> 1) You can take a look at PigUnit unility
> 2) Avro has a human-readable (json text, non-binary) form for debugging
> purposes
>
> Best Regards,
> Ruslan
>
>
> On Thu, Aug 29, 2013 at 2:56 PM, Serega Sheypak <[EMAIL PROTECTED]
> >wrote:
>
> > Hi, we have itegration test java utility which helps to test pig scripts.
> > We often develop different UDFs in java and I would like to create unit
> > tests for them. Right now they are tested with pig scripts during
> > integration tests.
> >
> > I have a pack of prepared avro files with etalon input for my java UDFs.
> > I would like to use some utility which would help me to convert avro to
> > tuple list.
> > I'm trying to use org.apache.pig.piggybank.storage.avro.AvroStorage with
> no
> > luck. I don't understand how it works.
> >
>
+
Ruslan Al-Fakikh 2013-08-30, 07:34
+
Serega Sheypak 2013-08-30, 08:18