Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Working with changing schemas (avro) in Pig


Copy link to this message
-
Re: Working with changing schemas (avro) in Pig
If you evolved your schema to just add fields, then you should be able to
use a single schema descriptor file to read both pre- and post-evolved data
objects. This is because one of the rules of new fields in Avro is that
they have to have a default value and be non-null. AvroStorage should pick
that default field up for the old objects. If it doesn't, then that's a bug.
On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:

> @Bill,
> I did look at the option of providing input as a parameter while
> initializing AvroStorage(). But even then, I'll still need to change my
> script to handle the two files because I'll still need to have separate
> schemas right?
>
> @Stan,
> Thanks for pointing me to it, it is a useful feature. But in my case, I
> would never have two input files with different schemas. The input will
> always have only one of the schemas, but I want my new script (with the
> additional column) to be able to process the old data as well, even if the
> input only contains data with the older schema.
>
> On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <[EMAIL PROTECTED]
> >wrote:
>
> > There is a patch for Avro to deal with this use case:
> > https://issues.apache.org/jira/browse/PIG-2579
> > (See the attached pig example which loads two avro input files with
> > different schemas.)
> >
> > Best,
> >
> > stan
> >
> > On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> > > Hi guys,
> > >
> > > I use Pig to process some clickstream data. I need to track a new
> field,
> > so
> > > I added a new field to my avro schema, and changed my Pig script
> > > accordingly. It works fine with the new files (which have that new
> > column)
> > > but it breaks when I run it on my old files which do not have that
> column
> > > in the schema (since avro stores schema in the data files itself). I
> was
> > > expecting that Pig will assume the field to be null if that particular
> > > field does not exist. But now I am having to maintain separate scripts
> to
> > > process the old and new files. Is there any workaround this? Because I
> > > figure I'll have to add new column frequently and I don't want to
> > maintain
> > > a separate script for each window where the schema is constant.
> > >
> > > Thanks,
> >
>

--
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*