IGZ Nick
2012-03-28, 20:22
Bill Graham
2012-03-28, 20:29
Stan Rosenberg
2012-03-28, 22:00
IGZ Nick
2012-03-28, 22:26
Bill Graham
2012-03-28, 22:53
IGZ Nick
2012-03-28, 23:01
Bill Graham
2012-03-28, 23:19
IGZ Nick
2012-03-29, 00:26
Bill Graham
2012-03-29, 00:41
IGZ Nick
2012-03-29, 20:05
Bill Graham
2012-03-30, 04:59
Alex Rovner
2012-04-01, 16:32
Bill Graham
2012-04-02, 05:08
Alex Rovner
2012-04-02, 17:18
Bill Graham
2012-04-02, 17:33
-
Working with changing schemas (avro) in PigIGZ Nick 2012-03-28, 20:22
Hi guys,
I use Pig to process some clickstream data. I need to track a new field, so
I added a new field to my avro schema, and changed my Pig script
accordingly. It works fine with the new files (which have that new column)
but it breaks when I run it on my old files which do not have that column
in the schema (since avro stores schema in the data files itself). I was
expecting that Pig will assume the field to be null if that particular
field does not exist. But now I am having to maintain separate scripts to
process the old and new files. Is there any workaround this? Because I
figure I'll have to add new column frequently and I don't want to maintain
a separate script for each window where the schema is constant.
Thanks,
I use Pig to process some clickstream data. I need to track a new field, so
I added a new field to my avro schema, and changed my Pig script
accordingly. It works fine with the new files (which have that new column)
but it breaks when I run it on my old files which do not have that column
in the schema (since avro stores schema in the data files itself). I was
expecting that Pig will assume the field to be null if that particular
field does not exist. But now I am having to maintain separate scripts to
process the old and new files. Is there any workaround this? Because I
figure I'll have to add new column frequently and I don't want to maintain
a separate script for each window where the schema is constant.
Thanks,
+
IGZ Nick 2012-03-28, 20:22
-
Re: Working with changing schemas (avro) in PigBill Graham 2012-03-28, 20:29
AvroStorage supports different modes to load the schema definition. One is
to get it from the Avro record, which would cause problems with evolution,
but you can also specific a schema file. Which are you using? Can you
attach the snippet of your script that initializes AvroStorage?
On Wed, Mar 28, 2012 at 1:22 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> Hi guys,
>
> I use Pig to process some clickstream data. I need to track a new field, so
> I added a new field to my avro schema, and changed my Pig script
> accordingly. It works fine with the new files (which have that new column)
> but it breaks when I run it on my old files which do not have that column
> in the schema (since avro stores schema in the data files itself). I was
> expecting that Pig will assume the field to be null if that particular
> field does not exist. But now I am having to maintain separate scripts to
> process the old and new files. Is there any workaround this? Because I
> figure I'll have to add new column frequently and I don't want to maintain
> a separate script for each window where the schema is constant.
>
> Thanks,
>
--
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*
to get it from the Avro record, which would cause problems with evolution,
but you can also specific a schema file. Which are you using? Can you
attach the snippet of your script that initializes AvroStorage?
On Wed, Mar 28, 2012 at 1:22 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> Hi guys,
>
> I use Pig to process some clickstream data. I need to track a new field, so
> I added a new field to my avro schema, and changed my Pig script
> accordingly. It works fine with the new files (which have that new column)
> but it breaks when I run it on my old files which do not have that column
> in the schema (since avro stores schema in the data files itself). I was
> expecting that Pig will assume the field to be null if that particular
> field does not exist. But now I am having to maintain separate scripts to
> process the old and new files. Is there any workaround this? Because I
> figure I'll have to add new column frequently and I don't want to maintain
> a separate script for each window where the schema is constant.
>
> Thanks,
>
--
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*
+
Bill Graham 2012-03-28, 20:29
-
Re: Working with changing schemas (avro) in PigStan Rosenberg 2012-03-28, 22:00
There is a patch for Avro to deal with this use case:
https://issues.apache.org/jira/browse/PIG-2579
(See the attached pig example which loads two avro input files with
different schemas.)
Best,
stan
On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> Hi guys,
>
> I use Pig to process some clickstream data. I need to track a new field, so
> I added a new field to my avro schema, and changed my Pig script
> accordingly. It works fine with the new files (which have that new column)
> but it breaks when I run it on my old files which do not have that column
> in the schema (since avro stores schema in the data files itself). I was
> expecting that Pig will assume the field to be null if that particular
> field does not exist. But now I am having to maintain separate scripts to
> process the old and new files. Is there any workaround this? Because I
> figure I'll have to add new column frequently and I don't want to maintain
> a separate script for each window where the schema is constant.
>
> Thanks,
https://issues.apache.org/jira/browse/PIG-2579
(See the attached pig example which loads two avro input files with
different schemas.)
Best,
stan
On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> Hi guys,
>
> I use Pig to process some clickstream data. I need to track a new field, so
> I added a new field to my avro schema, and changed my Pig script
> accordingly. It works fine with the new files (which have that new column)
> but it breaks when I run it on my old files which do not have that column
> in the schema (since avro stores schema in the data files itself). I was
> expecting that Pig will assume the field to be null if that particular
> field does not exist. But now I am having to maintain separate scripts to
> process the old and new files. Is there any workaround this? Because I
> figure I'll have to add new column frequently and I don't want to maintain
> a separate script for each window where the schema is constant.
>
> Thanks,
+
Stan Rosenberg 2012-03-28, 22:00
-
Re: Working with changing schemas (avro) in PigIGZ Nick 2012-03-28, 22:26
@Bill,
I did look at the option of providing input as a parameter while
initializing AvroStorage(). But even then, I'll still need to change my
script to handle the two files because I'll still need to have separate
schemas right?
@Stan,
Thanks for pointing me to it, it is a useful feature. But in my case, I
would never have two input files with different schemas. The input will
always have only one of the schemas, but I want my new script (with the
additional column) to be able to process the old data as well, even if the
input only contains data with the older schema.
On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <[EMAIL PROTECTED]>wrote:
> There is a patch for Avro to deal with this use case:
> https://issues.apache.org/jira/browse/PIG-2579
> (See the attached pig example which loads two avro input files with
> different schemas.)
>
> Best,
>
> stan
>
> On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> > Hi guys,
> >
> > I use Pig to process some clickstream data. I need to track a new field,
> so
> > I added a new field to my avro schema, and changed my Pig script
> > accordingly. It works fine with the new files (which have that new
> column)
> > but it breaks when I run it on my old files which do not have that column
> > in the schema (since avro stores schema in the data files itself). I was
> > expecting that Pig will assume the field to be null if that particular
> > field does not exist. But now I am having to maintain separate scripts to
> > process the old and new files. Is there any workaround this? Because I
> > figure I'll have to add new column frequently and I don't want to
> maintain
> > a separate script for each window where the schema is constant.
> >
> > Thanks,
>
I did look at the option of providing input as a parameter while
initializing AvroStorage(). But even then, I'll still need to change my
script to handle the two files because I'll still need to have separate
schemas right?
@Stan,
Thanks for pointing me to it, it is a useful feature. But in my case, I
would never have two input files with different schemas. The input will
always have only one of the schemas, but I want my new script (with the
additional column) to be able to process the old data as well, even if the
input only contains data with the older schema.
On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <[EMAIL PROTECTED]>wrote:
> There is a patch for Avro to deal with this use case:
> https://issues.apache.org/jira/browse/PIG-2579
> (See the attached pig example which loads two avro input files with
> different schemas.)
>
> Best,
>
> stan
>
> On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> > Hi guys,
> >
> > I use Pig to process some clickstream data. I need to track a new field,
> so
> > I added a new field to my avro schema, and changed my Pig script
> > accordingly. It works fine with the new files (which have that new
> column)
> > but it breaks when I run it on my old files which do not have that column
> > in the schema (since avro stores schema in the data files itself). I was
> > expecting that Pig will assume the field to be null if that particular
> > field does not exist. But now I am having to maintain separate scripts to
> > process the old and new files. Is there any workaround this? Because I
> > figure I'll have to add new column frequently and I don't want to
> maintain
> > a separate script for each window where the schema is constant.
> >
> > Thanks,
>
+
IGZ Nick 2012-03-28, 22:26
-
Re: Working with changing schemas (avro) in PigBill Graham 2012-03-28, 22:53
If you evolved your schema to just add fields, then you should be able to
use a single schema descriptor file to read both pre- and post-evolved data
objects. This is because one of the rules of new fields in Avro is that
they have to have a default value and be non-null. AvroStorage should pick
that default field up for the old objects. If it doesn't, then that's a bug.
On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> @Bill,
> I did look at the option of providing input as a parameter while
> initializing AvroStorage(). But even then, I'll still need to change my
> script to handle the two files because I'll still need to have separate
> schemas right?
>
> @Stan,
> Thanks for pointing me to it, it is a useful feature. But in my case, I
> would never have two input files with different schemas. The input will
> always have only one of the schemas, but I want my new script (with the
> additional column) to be able to process the old data as well, even if the
> input only contains data with the older schema.
>
> On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <[EMAIL PROTECTED]
> >wrote:
>
> > There is a patch for Avro to deal with this use case:
> > https://issues.apache.org/jira/browse/PIG-2579
> > (See the attached pig example which loads two avro input files with
> > different schemas.)
> >
> > Best,
> >
> > stan
> >
> > On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> > > Hi guys,
> > >
> > > I use Pig to process some clickstream data. I need to track a new
> field,
> > so
> > > I added a new field to my avro schema, and changed my Pig script
> > > accordingly. It works fine with the new files (which have that new
> > column)
> > > but it breaks when I run it on my old files which do not have that
> column
> > > in the schema (since avro stores schema in the data files itself). I
> was
> > > expecting that Pig will assume the field to be null if that particular
> > > field does not exist. But now I am having to maintain separate scripts
> to
> > > process the old and new files. Is there any workaround this? Because I
> > > figure I'll have to add new column frequently and I don't want to
> > maintain
> > > a separate script for each window where the schema is constant.
> > >
> > > Thanks,
> >
>
--
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*
use a single schema descriptor file to read both pre- and post-evolved data
objects. This is because one of the rules of new fields in Avro is that
they have to have a default value and be non-null. AvroStorage should pick
that default field up for the old objects. If it doesn't, then that's a bug.
On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> @Bill,
> I did look at the option of providing input as a parameter while
> initializing AvroStorage(). But even then, I'll still need to change my
> script to handle the two files because I'll still need to have separate
> schemas right?
>
> @Stan,
> Thanks for pointing me to it, it is a useful feature. But in my case, I
> would never have two input files with different schemas. The input will
> always have only one of the schemas, but I want my new script (with the
> additional column) to be able to process the old data as well, even if the
> input only contains data with the older schema.
>
> On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <[EMAIL PROTECTED]
> >wrote:
>
> > There is a patch for Avro to deal with this use case:
> > https://issues.apache.org/jira/browse/PIG-2579
> > (See the attached pig example which loads two avro input files with
> > different schemas.)
> >
> > Best,
> >
> > stan
> >
> > On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> > > Hi guys,
> > >
> > > I use Pig to process some clickstream data. I need to track a new
> field,
> > so
> > > I added a new field to my avro schema, and changed my Pig script
> > > accordingly. It works fine with the new files (which have that new
> > column)
> > > but it breaks when I run it on my old files which do not have that
> column
> > > in the schema (since avro stores schema in the data files itself). I
> was
> > > expecting that Pig will assume the field to be null if that particular
> > > field does not exist. But now I am having to maintain separate scripts
> to
> > > process the old and new files. Is there any workaround this? Because I
> > > figure I'll have to add new column frequently and I don't want to
> > maintain
> > > a separate script for each window where the schema is constant.
> > >
> > > Thanks,
> >
>
--
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*
+
Bill Graham 2012-03-28, 22:53
-
Re: Working with changing schemas (avro) in PigIGZ Nick 2012-03-28, 23:01
Ok.. you mean I can just use the newer schema to read the old schema as
well, by populating some default value for the missing field. I think that
should work, messy code though!
Thanks!
On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
> If you evolved your schema to just add fields, then you should be able to
> use a single schema descriptor file to read both pre- and post-evolved data
> objects. This is because one of the rules of new fields in Avro is that
> they have to have a default value and be non-null. AvroStorage should pick
> that default field up for the old objects. If it doesn't, then that's a
> bug.
>
>
> On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>
> > @Bill,
> > I did look at the option of providing input as a parameter while
> > initializing AvroStorage(). But even then, I'll still need to change my
> > script to handle the two files because I'll still need to have separate
> > schemas right?
> >
> > @Stan,
> > Thanks for pointing me to it, it is a useful feature. But in my case, I
> > would never have two input files with different schemas. The input will
> > always have only one of the schemas, but I want my new script (with the
> > additional column) to be able to process the old data as well, even if
> the
> > input only contains data with the older schema.
> >
> > On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <
> [EMAIL PROTECTED]
> > >wrote:
> >
> > > There is a patch for Avro to deal with this use case:
> > > https://issues.apache.org/jira/browse/PIG-2579
> > > (See the attached pig example which loads two avro input files with
> > > different schemas.)
> > >
> > > Best,
> > >
> > > stan
> > >
> > > On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> > > > Hi guys,
> > > >
> > > > I use Pig to process some clickstream data. I need to track a new
> > field,
> > > so
> > > > I added a new field to my avro schema, and changed my Pig script
> > > > accordingly. It works fine with the new files (which have that new
> > > column)
> > > > but it breaks when I run it on my old files which do not have that
> > column
> > > > in the schema (since avro stores schema in the data files itself). I
> > was
> > > > expecting that Pig will assume the field to be null if that
> particular
> > > > field does not exist. But now I am having to maintain separate
> scripts
> > to
> > > > process the old and new files. Is there any workaround this? Because
> I
> > > > figure I'll have to add new column frequently and I don't want to
> > > maintain
> > > > a separate script for each window where the schema is constant.
> > > >
> > > > Thanks,
> > >
> >
>
>
>
> --
> *Note that I'm no longer using my Yahoo! email address. Please email me at
> [EMAIL PROTECTED] going forward.*
>
well, by populating some default value for the missing field. I think that
should work, messy code though!
Thanks!
On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
> If you evolved your schema to just add fields, then you should be able to
> use a single schema descriptor file to read both pre- and post-evolved data
> objects. This is because one of the rules of new fields in Avro is that
> they have to have a default value and be non-null. AvroStorage should pick
> that default field up for the old objects. If it doesn't, then that's a
> bug.
>
>
> On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>
> > @Bill,
> > I did look at the option of providing input as a parameter while
> > initializing AvroStorage(). But even then, I'll still need to change my
> > script to handle the two files because I'll still need to have separate
> > schemas right?
> >
> > @Stan,
> > Thanks for pointing me to it, it is a useful feature. But in my case, I
> > would never have two input files with different schemas. The input will
> > always have only one of the schemas, but I want my new script (with the
> > additional column) to be able to process the old data as well, even if
> the
> > input only contains data with the older schema.
> >
> > On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <
> [EMAIL PROTECTED]
> > >wrote:
> >
> > > There is a patch for Avro to deal with this use case:
> > > https://issues.apache.org/jira/browse/PIG-2579
> > > (See the attached pig example which loads two avro input files with
> > > different schemas.)
> > >
> > > Best,
> > >
> > > stan
> > >
> > > On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> > > > Hi guys,
> > > >
> > > > I use Pig to process some clickstream data. I need to track a new
> > field,
> > > so
> > > > I added a new field to my avro schema, and changed my Pig script
> > > > accordingly. It works fine with the new files (which have that new
> > > column)
> > > > but it breaks when I run it on my old files which do not have that
> > column
> > > > in the schema (since avro stores schema in the data files itself). I
> > was
> > > > expecting that Pig will assume the field to be null if that
> particular
> > > > field does not exist. But now I am having to maintain separate
> scripts
> > to
> > > > process the old and new files. Is there any workaround this? Because
> I
> > > > figure I'll have to add new column frequently and I don't want to
> > > maintain
> > > > a separate script for each window where the schema is constant.
> > > >
> > > > Thanks,
> > >
> >
>
>
>
> --
> *Note that I'm no longer using my Yahoo! email address. Please email me at
> [EMAIL PROTECTED] going forward.*
>
+
IGZ Nick 2012-03-28, 23:01
-
Re: Working with changing schemas (avro) in PigBill Graham 2012-03-28, 23:19
The default value will be part of the new Avro schema definition and Avro
should return it to you, so there shouldn't be any code messyness with that
approach.
On Wed, Mar 28, 2012 at 4:01 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> Ok.. you mean I can just use the newer schema to read the old schema as
> well, by populating some default value for the missing field. I think that
> should work, messy code though!
>
> Thanks!
>
> On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
>> If you evolved your schema to just add fields, then you should be able to
>> use a single schema descriptor file to read both pre- and post-evolved
>> data
>> objects. This is because one of the rules of new fields in Avro is that
>> they have to have a default value and be non-null. AvroStorage should pick
>> that default field up for the old objects. If it doesn't, then that's a
>> bug.
>>
>>
>> On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>
>> > @Bill,
>> > I did look at the option of providing input as a parameter while
>> > initializing AvroStorage(). But even then, I'll still need to change my
>> > script to handle the two files because I'll still need to have separate
>> > schemas right?
>> >
>> > @Stan,
>> > Thanks for pointing me to it, it is a useful feature. But in my case, I
>> > would never have two input files with different schemas. The input will
>> > always have only one of the schemas, but I want my new script (with the
>> > additional column) to be able to process the old data as well, even if
>> the
>> > input only contains data with the older schema.
>> >
>> > On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <
>> [EMAIL PROTECTED]
>> > >wrote:
>> >
>> > > There is a patch for Avro to deal with this use case:
>> > > https://issues.apache.org/jira/browse/PIG-2579
>> > > (See the attached pig example which loads two avro input files with
>> > > different schemas.)
>> > >
>> > > Best,
>> > >
>> > > stan
>> > >
>> > > On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]>
>> wrote:
>> > > > Hi guys,
>> > > >
>> > > > I use Pig to process some clickstream data. I need to track a new
>> > field,
>> > > so
>> > > > I added a new field to my avro schema, and changed my Pig script
>> > > > accordingly. It works fine with the new files (which have that new
>> > > column)
>> > > > but it breaks when I run it on my old files which do not have that
>> > column
>> > > > in the schema (since avro stores schema in the data files itself). I
>> > was
>> > > > expecting that Pig will assume the field to be null if that
>> particular
>> > > > field does not exist. But now I am having to maintain separate
>> scripts
>> > to
>> > > > process the old and new files. Is there any workaround this?
>> Because I
>> > > > figure I'll have to add new column frequently and I don't want to
>> > > maintain
>> > > > a separate script for each window where the schema is constant.
>> > > >
>> > > > Thanks,
>> > >
>> >
>>
>>
>>
>> --
>> *Note that I'm no longer using my Yahoo! email address. Please email me at
>> [EMAIL PROTECTED] going forward.*
>>
>
>
--
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*
should return it to you, so there shouldn't be any code messyness with that
approach.
On Wed, Mar 28, 2012 at 4:01 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> Ok.. you mean I can just use the newer schema to read the old schema as
> well, by populating some default value for the missing field. I think that
> should work, messy code though!
>
> Thanks!
>
> On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
>> If you evolved your schema to just add fields, then you should be able to
>> use a single schema descriptor file to read both pre- and post-evolved
>> data
>> objects. This is because one of the rules of new fields in Avro is that
>> they have to have a default value and be non-null. AvroStorage should pick
>> that default field up for the old objects. If it doesn't, then that's a
>> bug.
>>
>>
>> On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>
>> > @Bill,
>> > I did look at the option of providing input as a parameter while
>> > initializing AvroStorage(). But even then, I'll still need to change my
>> > script to handle the two files because I'll still need to have separate
>> > schemas right?
>> >
>> > @Stan,
>> > Thanks for pointing me to it, it is a useful feature. But in my case, I
>> > would never have two input files with different schemas. The input will
>> > always have only one of the schemas, but I want my new script (with the
>> > additional column) to be able to process the old data as well, even if
>> the
>> > input only contains data with the older schema.
>> >
>> > On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <
>> [EMAIL PROTECTED]
>> > >wrote:
>> >
>> > > There is a patch for Avro to deal with this use case:
>> > > https://issues.apache.org/jira/browse/PIG-2579
>> > > (See the attached pig example which loads two avro input files with
>> > > different schemas.)
>> > >
>> > > Best,
>> > >
>> > > stan
>> > >
>> > > On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]>
>> wrote:
>> > > > Hi guys,
>> > > >
>> > > > I use Pig to process some clickstream data. I need to track a new
>> > field,
>> > > so
>> > > > I added a new field to my avro schema, and changed my Pig script
>> > > > accordingly. It works fine with the new files (which have that new
>> > > column)
>> > > > but it breaks when I run it on my old files which do not have that
>> > column
>> > > > in the schema (since avro stores schema in the data files itself). I
>> > was
>> > > > expecting that Pig will assume the field to be null if that
>> particular
>> > > > field does not exist. But now I am having to maintain separate
>> scripts
>> > to
>> > > > process the old and new files. Is there any workaround this?
>> Because I
>> > > > figure I'll have to add new column frequently and I don't want to
>> > > maintain
>> > > > a separate script for each window where the schema is constant.
>> > > >
>> > > > Thanks,
>> > >
>> >
>>
>>
>>
>> --
>> *Note that I'm no longer using my Yahoo! email address. Please email me at
>> [EMAIL PROTECTED] going forward.*
>>
>
>
--
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*
+
Bill Graham 2012-03-28, 23:19
-
Re: Working with changing schemas (avro) in PigIGZ Nick 2012-03-29, 00:26
The schema has to be written in the script right? I don't think there is
any way the schema can be in a file outside the script. That was the
messyness I was talking about. Or is there a way I can write the schema in
a separate file? One way I see is to create and store a dummy file with the
schema
Wed, Mar 28, 2012 at 4:19 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
> The default value will be part of the new Avro schema definition and Avro
> should return it to you, so there shouldn't be any code messyness with that
> approach.
>
>
> On Wed, Mar 28, 2012 at 4:01 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>
>> Ok.. you mean I can just use the newer schema to read the old schema as
>> well, by populating some default value for the missing field. I think that
>> should work, messy code though!
>>
>> Thanks!
>>
>> On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]>wrote:
>>
>>> If you evolved your schema to just add fields, then you should be able
>>> to
>>> use a single schema descriptor file to read both pre- and post-evolved
>>> data
>>> objects. This is because one of the rules of new fields in Avro is that
>>> they have to have a default value and be non-null. AvroStorage should
>>> pick
>>> that default field up for the old objects. If it doesn't, then that's a
>>> bug.
>>>
>>>
>>> On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>>
>>> > @Bill,
>>> > I did look at the option of providing input as a parameter while
>>> > initializing AvroStorage(). But even then, I'll still need to change my
>>> > script to handle the two files because I'll still need to have separate
>>> > schemas right?
>>> >
>>> > @Stan,
>>> > Thanks for pointing me to it, it is a useful feature. But in my case, I
>>> > would never have two input files with different schemas. The input will
>>> > always have only one of the schemas, but I want my new script (with the
>>> > additional column) to be able to process the old data as well, even if
>>> the
>>> > input only contains data with the older schema.
>>> >
>>> > On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <
>>> [EMAIL PROTECTED]
>>> > >wrote:
>>> >
>>> > > There is a patch for Avro to deal with this use case:
>>> > > https://issues.apache.org/jira/browse/PIG-2579
>>> > > (See the attached pig example which loads two avro input files with
>>> > > different schemas.)
>>> > >
>>> > > Best,
>>> > >
>>> > > stan
>>> > >
>>> > > On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]>
>>> wrote:
>>> > > > Hi guys,
>>> > > >
>>> > > > I use Pig to process some clickstream data. I need to track a new
>>> > field,
>>> > > so
>>> > > > I added a new field to my avro schema, and changed my Pig script
>>> > > > accordingly. It works fine with the new files (which have that new
>>> > > column)
>>> > > > but it breaks when I run it on my old files which do not have that
>>> > column
>>> > > > in the schema (since avro stores schema in the data files itself).
>>> I
>>> > was
>>> > > > expecting that Pig will assume the field to be null if that
>>> particular
>>> > > > field does not exist. But now I am having to maintain separate
>>> scripts
>>> > to
>>> > > > process the old and new files. Is there any workaround this?
>>> Because I
>>> > > > figure I'll have to add new column frequently and I don't want to
>>> > > maintain
>>> > > > a separate script for each window where the schema is constant.
>>> > > >
>>> > > > Thanks,
>>> > >
>>> >
>>>
>>>
>>>
>>> --
>>> *Note that I'm no longer using my Yahoo! email address. Please email me
>>> at
>>> [EMAIL PROTECTED] going forward.*
>>>
>>
>>
>
>
> --
> *Note that I'm no longer using my Yahoo! email address. Please email me
> at [EMAIL PROTECTED] going forward.*
>
any way the schema can be in a file outside the script. That was the
messyness I was talking about. Or is there a way I can write the schema in
a separate file? One way I see is to create and store a dummy file with the
schema
Wed, Mar 28, 2012 at 4:19 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
> The default value will be part of the new Avro schema definition and Avro
> should return it to you, so there shouldn't be any code messyness with that
> approach.
>
>
> On Wed, Mar 28, 2012 at 4:01 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>
>> Ok.. you mean I can just use the newer schema to read the old schema as
>> well, by populating some default value for the missing field. I think that
>> should work, messy code though!
>>
>> Thanks!
>>
>> On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]>wrote:
>>
>>> If you evolved your schema to just add fields, then you should be able
>>> to
>>> use a single schema descriptor file to read both pre- and post-evolved
>>> data
>>> objects. This is because one of the rules of new fields in Avro is that
>>> they have to have a default value and be non-null. AvroStorage should
>>> pick
>>> that default field up for the old objects. If it doesn't, then that's a
>>> bug.
>>>
>>>
>>> On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>>
>>> > @Bill,
>>> > I did look at the option of providing input as a parameter while
>>> > initializing AvroStorage(). But even then, I'll still need to change my
>>> > script to handle the two files because I'll still need to have separate
>>> > schemas right?
>>> >
>>> > @Stan,
>>> > Thanks for pointing me to it, it is a useful feature. But in my case, I
>>> > would never have two input files with different schemas. The input will
>>> > always have only one of the schemas, but I want my new script (with the
>>> > additional column) to be able to process the old data as well, even if
>>> the
>>> > input only contains data with the older schema.
>>> >
>>> > On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <
>>> [EMAIL PROTECTED]
>>> > >wrote:
>>> >
>>> > > There is a patch for Avro to deal with this use case:
>>> > > https://issues.apache.org/jira/browse/PIG-2579
>>> > > (See the attached pig example which loads two avro input files with
>>> > > different schemas.)
>>> > >
>>> > > Best,
>>> > >
>>> > > stan
>>> > >
>>> > > On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]>
>>> wrote:
>>> > > > Hi guys,
>>> > > >
>>> > > > I use Pig to process some clickstream data. I need to track a new
>>> > field,
>>> > > so
>>> > > > I added a new field to my avro schema, and changed my Pig script
>>> > > > accordingly. It works fine with the new files (which have that new
>>> > > column)
>>> > > > but it breaks when I run it on my old files which do not have that
>>> > column
>>> > > > in the schema (since avro stores schema in the data files itself).
>>> I
>>> > was
>>> > > > expecting that Pig will assume the field to be null if that
>>> particular
>>> > > > field does not exist. But now I am having to maintain separate
>>> scripts
>>> > to
>>> > > > process the old and new files. Is there any workaround this?
>>> Because I
>>> > > > figure I'll have to add new column frequently and I don't want to
>>> > > maintain
>>> > > > a separate script for each window where the schema is constant.
>>> > > >
>>> > > > Thanks,
>>> > >
>>> >
>>>
>>>
>>>
>>> --
>>> *Note that I'm no longer using my Yahoo! email address. Please email me
>>> at
>>> [EMAIL PROTECTED] going forward.*
>>>
>>
>>
>
>
> --
> *Note that I'm no longer using my Yahoo! email address. Please email me
> at [EMAIL PROTECTED] going forward.*
>
+
IGZ Nick 2012-03-29, 00:26
-
Re: Working with changing schemas (avro) in PigBill Graham 2012-03-29, 00:41
Yes, the schema can be in HDFS but the documentation for this is lacking.
Search for 'schema_file' here:
http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
and here:
http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
And be aware of this open JIRA:
https://issues.apache.org/jira/browse/PIG-2257
And this closed one:
https://issues.apache.org/jira/browse/PIG-2195
:)
thanks,
Bill
On Wed, Mar 28, 2012 at 5:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> The schema has to be written in the script right? I don't think there is
> any way the schema can be in a file outside the script. That was the
> messyness I was talking about. Or is there a way I can write the schema in
> a separate file? One way I see is to create and store a dummy file with the
> schema
>
>
> Wed, Mar 28, 2012 at 4:19 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
>> The default value will be part of the new Avro schema definition and Avro
>> should return it to you, so there shouldn't be any code messyness with that
>> approach.
>>
>>
>> On Wed, Mar 28, 2012 at 4:01 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>
>>> Ok.. you mean I can just use the newer schema to read the old schema as
>>> well, by populating some default value for the missing field. I think that
>>> should work, messy code though!
>>>
>>> Thanks!
>>>
>>> On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]>wrote:
>>>
>>>> If you evolved your schema to just add fields, then you should be able
>>>> to
>>>> use a single schema descriptor file to read both pre- and post-evolved
>>>> data
>>>> objects. This is because one of the rules of new fields in Avro is that
>>>> they have to have a default value and be non-null. AvroStorage should
>>>> pick
>>>> that default field up for the old objects. If it doesn't, then that's a
>>>> bug.
>>>>
>>>>
>>>> On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>>>
>>>> > @Bill,
>>>> > I did look at the option of providing input as a parameter while
>>>> > initializing AvroStorage(). But even then, I'll still need to change
>>>> my
>>>> > script to handle the two files because I'll still need to have
>>>> separate
>>>> > schemas right?
>>>> >
>>>> > @Stan,
>>>> > Thanks for pointing me to it, it is a useful feature. But in my case,
>>>> I
>>>> > would never have two input files with different schemas. The input
>>>> will
>>>> > always have only one of the schemas, but I want my new script (with
>>>> the
>>>> > additional column) to be able to process the old data as well, even
>>>> if the
>>>> > input only contains data with the older schema.
>>>> >
>>>> > On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <
>>>> [EMAIL PROTECTED]
>>>> > >wrote:
>>>> >
>>>> > > There is a patch for Avro to deal with this use case:
>>>> > > https://issues.apache.org/jira/browse/PIG-2579
>>>> > > (See the attached pig example which loads two avro input files with
>>>> > > different schemas.)
>>>> > >
>>>> > > Best,
>>>> > >
>>>> > > stan
>>>> > >
>>>> > > On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]>
>>>> wrote:
>>>> > > > Hi guys,
>>>> > > >
>>>> > > > I use Pig to process some clickstream data. I need to track a new
>>>> > field,
>>>> > > so
>>>> > > > I added a new field to my avro schema, and changed my Pig script
>>>> > > > accordingly. It works fine with the new files (which have that new
>>>> > > column)
>>>> > > > but it breaks when I run it on my old files which do not have that
>>>> > column
>>>> > > > in the schema (since avro stores schema in the data files
>>>> itself). I
>>>> > was
>>>> > > > expecting that Pig will assume the field to be null if that
>>>> particular
>>>> > > > field does not exist. But now I am having to maintain separate
>>>> scripts
>>>> > to
>>>> > > > process the old and new files. Is there any workaround this?
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*
Search for 'schema_file' here:
http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
and here:
http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
And be aware of this open JIRA:
https://issues.apache.org/jira/browse/PIG-2257
And this closed one:
https://issues.apache.org/jira/browse/PIG-2195
:)
thanks,
Bill
On Wed, Mar 28, 2012 at 5:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> The schema has to be written in the script right? I don't think there is
> any way the schema can be in a file outside the script. That was the
> messyness I was talking about. Or is there a way I can write the schema in
> a separate file? One way I see is to create and store a dummy file with the
> schema
>
>
> Wed, Mar 28, 2012 at 4:19 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
>> The default value will be part of the new Avro schema definition and Avro
>> should return it to you, so there shouldn't be any code messyness with that
>> approach.
>>
>>
>> On Wed, Mar 28, 2012 at 4:01 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>
>>> Ok.. you mean I can just use the newer schema to read the old schema as
>>> well, by populating some default value for the missing field. I think that
>>> should work, messy code though!
>>>
>>> Thanks!
>>>
>>> On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]>wrote:
>>>
>>>> If you evolved your schema to just add fields, then you should be able
>>>> to
>>>> use a single schema descriptor file to read both pre- and post-evolved
>>>> data
>>>> objects. This is because one of the rules of new fields in Avro is that
>>>> they have to have a default value and be non-null. AvroStorage should
>>>> pick
>>>> that default field up for the old objects. If it doesn't, then that's a
>>>> bug.
>>>>
>>>>
>>>> On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>>>
>>>> > @Bill,
>>>> > I did look at the option of providing input as a parameter while
>>>> > initializing AvroStorage(). But even then, I'll still need to change
>>>> my
>>>> > script to handle the two files because I'll still need to have
>>>> separate
>>>> > schemas right?
>>>> >
>>>> > @Stan,
>>>> > Thanks for pointing me to it, it is a useful feature. But in my case,
>>>> I
>>>> > would never have two input files with different schemas. The input
>>>> will
>>>> > always have only one of the schemas, but I want my new script (with
>>>> the
>>>> > additional column) to be able to process the old data as well, even
>>>> if the
>>>> > input only contains data with the older schema.
>>>> >
>>>> > On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <
>>>> [EMAIL PROTECTED]
>>>> > >wrote:
>>>> >
>>>> > > There is a patch for Avro to deal with this use case:
>>>> > > https://issues.apache.org/jira/browse/PIG-2579
>>>> > > (See the attached pig example which loads two avro input files with
>>>> > > different schemas.)
>>>> > >
>>>> > > Best,
>>>> > >
>>>> > > stan
>>>> > >
>>>> > > On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]>
>>>> wrote:
>>>> > > > Hi guys,
>>>> > > >
>>>> > > > I use Pig to process some clickstream data. I need to track a new
>>>> > field,
>>>> > > so
>>>> > > > I added a new field to my avro schema, and changed my Pig script
>>>> > > > accordingly. It works fine with the new files (which have that new
>>>> > > column)
>>>> > > > but it breaks when I run it on my old files which do not have that
>>>> > column
>>>> > > > in the schema (since avro stores schema in the data files
>>>> itself). I
>>>> > was
>>>> > > > expecting that Pig will assume the field to be null if that
>>>> particular
>>>> > > > field does not exist. But now I am having to maintain separate
>>>> scripts
>>>> > to
>>>> > > > process the old and new files. Is there any workaround this?
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*
+
Bill Graham 2012-03-29, 00:41
-
Re: Working with changing schemas (avro) in PigIGZ Nick 2012-03-29, 20:05
That's nice! Can you give me an example of how to use it? I am not able to
figure it out from the code. The schemaManager is only used at one place
after that, and that is when the params contains a "field<number>" key. I
don't understand that part. Is there a way I can call it simply like STORE
xyz INTO 'abc' USING AvroStorage('schema_file=/path/to/schema/file')?
On Wed, Mar 28, 2012 at 5:41 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
> Yes, the schema can be in HDFS but the documentation for this is lacking.
> Search for 'schema_file' here:
>
>
> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
>
> and here:
>
>
> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
>
> And be aware of this open JIRA:
> https://issues.apache.org/jira/browse/PIG-2257
>
> And this closed one:
> https://issues.apache.org/jira/browse/PIG-2195
>
> :)
>
> thanks,
> Bill
>
>
> On Wed, Mar 28, 2012 at 5:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>
>> The schema has to be written in the script right? I don't think there is
>> any way the schema can be in a file outside the script. That was the
>> messyness I was talking about. Or is there a way I can write the schema in
>> a separate file? One way I see is to create and store a dummy file with the
>> schema
>>
>>
>> Wed, Mar 28, 2012 at 4:19 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>>
>>> The default value will be part of the new Avro schema definition and
>>> Avro should return it to you, so there shouldn't be any code messyness with
>>> that approach.
>>>
>>>
>>> On Wed, Mar 28, 2012 at 4:01 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>>
>>>> Ok.. you mean I can just use the newer schema to read the old schema as
>>>> well, by populating some default value for the missing field. I think that
>>>> should work, messy code though!
>>>>
>>>> Thanks!
>>>>
>>>> On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> If you evolved your schema to just add fields, then you should be
>>>>> able to
>>>>> use a single schema descriptor file to read both pre- and post-evolved
>>>>> data
>>>>> objects. This is because one of the rules of new fields in Avro is that
>>>>> they have to have a default value and be non-null. AvroStorage should
>>>>> pick
>>>>> that default field up for the old objects. If it doesn't, then that's
>>>>> a bug.
>>>>>
>>>>>
>>>>> On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>> > @Bill,
>>>>> > I did look at the option of providing input as a parameter while
>>>>> > initializing AvroStorage(). But even then, I'll still need to change
>>>>> my
>>>>> > script to handle the two files because I'll still need to have
>>>>> separate
>>>>> > schemas right?
>>>>> >
>>>>> > @Stan,
>>>>> > Thanks for pointing me to it, it is a useful feature. But in my
>>>>> case, I
>>>>> > would never have two input files with different schemas. The input
>>>>> will
>>>>> > always have only one of the schemas, but I want my new script (with
>>>>> the
>>>>> > additional column) to be able to process the old data as well, even
>>>>> if the
>>>>> > input only contains data with the older schema.
>>>>> >
>>>>> > On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <
>>>>> [EMAIL PROTECTED]
>>>>> > >wrote:
>>>>> >
>>>>> > > There is a patch for Avro to deal with this use case:
>>>>> > > https://issues.apache.org/jira/browse/PIG-2579
>>>>> > > (See the attached pig example which loads two avro input files with
>>>>> > > different schemas.)
>>>>> > >
>>>>> > > Best,
>>>>> > >
>>>>> > > stan
>>>>> > >
>>>>> > > On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>> > > > Hi guys,
>>>>> > > >
>>>>> > > > I use Pig to process some clickstream data. I need to track a new
>>>>> > field,
>>>>> > > so
>>>>> > > > I added a new field to my avro schema, and changed my Pig script
figure it out from the code. The schemaManager is only used at one place
after that, and that is when the params contains a "field<number>" key. I
don't understand that part. Is there a way I can call it simply like STORE
xyz INTO 'abc' USING AvroStorage('schema_file=/path/to/schema/file')?
On Wed, Mar 28, 2012 at 5:41 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
> Yes, the schema can be in HDFS but the documentation for this is lacking.
> Search for 'schema_file' here:
>
>
> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
>
> and here:
>
>
> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
>
> And be aware of this open JIRA:
> https://issues.apache.org/jira/browse/PIG-2257
>
> And this closed one:
> https://issues.apache.org/jira/browse/PIG-2195
>
> :)
>
> thanks,
> Bill
>
>
> On Wed, Mar 28, 2012 at 5:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>
>> The schema has to be written in the script right? I don't think there is
>> any way the schema can be in a file outside the script. That was the
>> messyness I was talking about. Or is there a way I can write the schema in
>> a separate file? One way I see is to create and store a dummy file with the
>> schema
>>
>>
>> Wed, Mar 28, 2012 at 4:19 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>>
>>> The default value will be part of the new Avro schema definition and
>>> Avro should return it to you, so there shouldn't be any code messyness with
>>> that approach.
>>>
>>>
>>> On Wed, Mar 28, 2012 at 4:01 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>>
>>>> Ok.. you mean I can just use the newer schema to read the old schema as
>>>> well, by populating some default value for the missing field. I think that
>>>> should work, messy code though!
>>>>
>>>> Thanks!
>>>>
>>>> On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> If you evolved your schema to just add fields, then you should be
>>>>> able to
>>>>> use a single schema descriptor file to read both pre- and post-evolved
>>>>> data
>>>>> objects. This is because one of the rules of new fields in Avro is that
>>>>> they have to have a default value and be non-null. AvroStorage should
>>>>> pick
>>>>> that default field up for the old objects. If it doesn't, then that's
>>>>> a bug.
>>>>>
>>>>>
>>>>> On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>> > @Bill,
>>>>> > I did look at the option of providing input as a parameter while
>>>>> > initializing AvroStorage(). But even then, I'll still need to change
>>>>> my
>>>>> > script to handle the two files because I'll still need to have
>>>>> separate
>>>>> > schemas right?
>>>>> >
>>>>> > @Stan,
>>>>> > Thanks for pointing me to it, it is a useful feature. But in my
>>>>> case, I
>>>>> > would never have two input files with different schemas. The input
>>>>> will
>>>>> > always have only one of the schemas, but I want my new script (with
>>>>> the
>>>>> > additional column) to be able to process the old data as well, even
>>>>> if the
>>>>> > input only contains data with the older schema.
>>>>> >
>>>>> > On Wed, Mar 28, 2012 at 3:00 PM, Stan Rosenberg <
>>>>> [EMAIL PROTECTED]
>>>>> > >wrote:
>>>>> >
>>>>> > > There is a patch for Avro to deal with this use case:
>>>>> > > https://issues.apache.org/jira/browse/PIG-2579
>>>>> > > (See the attached pig example which loads two avro input files with
>>>>> > > different schemas.)
>>>>> > >
>>>>> > > Best,
>>>>> > >
>>>>> > > stan
>>>>> > >
>>>>> > > On Wed, Mar 28, 2012 at 4:22 PM, IGZ Nick <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>> > > > Hi guys,
>>>>> > > >
>>>>> > > > I use Pig to process some clickstream data. I need to track a new
>>>>> > field,
>>>>> > > so
>>>>> > > > I added a new field to my avro schema, and changed my Pig script
+
IGZ Nick 2012-03-29, 20:05
-
Re: Working with changing schemas (avro) in PigBill Graham 2012-03-30, 04:59
In the TestAvroStorage.testRecordWithFieldSchemaFromTextWithSchemaFile
there's an example:
STORE avro2 INTO 'output_dir'
USING org.apache.pig.piggybank.storage.avro.AvroStorage (
'{"schema_file": "/path/to/schema/file" ,
"field0": "def:member_id",
"field1": "def:browser_id",
"field3": "def:act_content" }'
);
You specify the file that contains the schema, then you have to map the
tuple fields to the name of the field in the avro schema. This mapping is a
drag, but it's currently required.
Note that only the json-style constructor (as opposed to the string array
appoach) supports schema_file without this uncommitted patch:
https://issues.apache.org/jira/browse/PIG-2257
thanks,
Bill
On Thu, Mar 29, 2012 at 1:05 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> That's nice! Can you give me an example of how to use it? I am not able to
> figure it out from the code. The schemaManager is only used at one place
> after that, and that is when the params contains a "field<number>" key. I
> don't understand that part. Is there a way I can call it simply like STORE
> xyz INTO 'abc' USING AvroStorage('schema_file=/path/to/schema/file')?
>
>
>
> On Wed, Mar 28, 2012 at 5:41 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
>> Yes, the schema can be in HDFS but the documentation for this is lacking.
>> Search for 'schema_file' here:
>>
>>
>> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
>>
>> and here:
>>
>>
>> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
>>
>> And be aware of this open JIRA:
>> https://issues.apache.org/jira/browse/PIG-2257
>>
>> And this closed one:
>> https://issues.apache.org/jira/browse/PIG-2195
>>
>> :)
>>
>> thanks,
>> Bill
>>
>>
>> On Wed, Mar 28, 2012 at 5:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>
>>> The schema has to be written in the script right? I don't think there is
>>> any way the schema can be in a file outside the script. That was the
>>> messyness I was talking about. Or is there a way I can write the schema in
>>> a separate file? One way I see is to create and store a dummy file with the
>>> schema
>>>
>>>
>>> Wed, Mar 28, 2012 at 4:19 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>>>
>>>> The default value will be part of the new Avro schema definition and
>>>> Avro should return it to you, so there shouldn't be any code messyness with
>>>> that approach.
>>>>
>>>>
>>>> On Wed, Mar 28, 2012 at 4:01 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Ok.. you mean I can just use the newer schema to read the old schema
>>>>> as well, by populating some default value for the missing field. I think
>>>>> that should work, messy code though!
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>> If you evolved your schema to just add fields, then you should be
>>>>>> able to
>>>>>> use a single schema descriptor file to read both pre- and
>>>>>> post-evolved data
>>>>>> objects. This is because one of the rules of new fields in Avro is
>>>>>> that
>>>>>> they have to have a default value and be non-null. AvroStorage should
>>>>>> pick
>>>>>> that default field up for the old objects. If it doesn't, then that's
>>>>>> a bug.
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]>
>>>>>> wrote:
>>>>>>
>>>>>> > @Bill,
>>>>>> > I did look at the option of providing input as a parameter while
>>>>>> > initializing AvroStorage(). But even then, I'll still need to
>>>>>> change my
>>>>>> > script to handle the two files because I'll still need to have
>>>>>> separate
>>>>>> > schemas right?
>>>>>> >
>>>>>> > @Stan,
>>>>>> > Thanks for pointing me to it, it is a useful feature. But in my
>>>>>> case, I
>>>>>> > would never have two input files with different schemas. The input
>>>>>> will
>>>>>> > always have only one of the schemas, but I want my new script (with
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*
there's an example:
STORE avro2 INTO 'output_dir'
USING org.apache.pig.piggybank.storage.avro.AvroStorage (
'{"schema_file": "/path/to/schema/file" ,
"field0": "def:member_id",
"field1": "def:browser_id",
"field3": "def:act_content" }'
);
You specify the file that contains the schema, then you have to map the
tuple fields to the name of the field in the avro schema. This mapping is a
drag, but it's currently required.
Note that only the json-style constructor (as opposed to the string array
appoach) supports schema_file without this uncommitted patch:
https://issues.apache.org/jira/browse/PIG-2257
thanks,
Bill
On Thu, Mar 29, 2012 at 1:05 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> That's nice! Can you give me an example of how to use it? I am not able to
> figure it out from the code. The schemaManager is only used at one place
> after that, and that is when the params contains a "field<number>" key. I
> don't understand that part. Is there a way I can call it simply like STORE
> xyz INTO 'abc' USING AvroStorage('schema_file=/path/to/schema/file')?
>
>
>
> On Wed, Mar 28, 2012 at 5:41 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
>> Yes, the schema can be in HDFS but the documentation for this is lacking.
>> Search for 'schema_file' here:
>>
>>
>> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
>>
>> and here:
>>
>>
>> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
>>
>> And be aware of this open JIRA:
>> https://issues.apache.org/jira/browse/PIG-2257
>>
>> And this closed one:
>> https://issues.apache.org/jira/browse/PIG-2195
>>
>> :)
>>
>> thanks,
>> Bill
>>
>>
>> On Wed, Mar 28, 2012 at 5:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>
>>> The schema has to be written in the script right? I don't think there is
>>> any way the schema can be in a file outside the script. That was the
>>> messyness I was talking about. Or is there a way I can write the schema in
>>> a separate file? One way I see is to create and store a dummy file with the
>>> schema
>>>
>>>
>>> Wed, Mar 28, 2012 at 4:19 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>>>
>>>> The default value will be part of the new Avro schema definition and
>>>> Avro should return it to you, so there shouldn't be any code messyness with
>>>> that approach.
>>>>
>>>>
>>>> On Wed, Mar 28, 2012 at 4:01 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Ok.. you mean I can just use the newer schema to read the old schema
>>>>> as well, by populating some default value for the missing field. I think
>>>>> that should work, messy code though!
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>> If you evolved your schema to just add fields, then you should be
>>>>>> able to
>>>>>> use a single schema descriptor file to read both pre- and
>>>>>> post-evolved data
>>>>>> objects. This is because one of the rules of new fields in Avro is
>>>>>> that
>>>>>> they have to have a default value and be non-null. AvroStorage should
>>>>>> pick
>>>>>> that default field up for the old objects. If it doesn't, then that's
>>>>>> a bug.
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]>
>>>>>> wrote:
>>>>>>
>>>>>> > @Bill,
>>>>>> > I did look at the option of providing input as a parameter while
>>>>>> > initializing AvroStorage(). But even then, I'll still need to
>>>>>> change my
>>>>>> > script to handle the two files because I'll still need to have
>>>>>> separate
>>>>>> > schemas right?
>>>>>> >
>>>>>> > @Stan,
>>>>>> > Thanks for pointing me to it, it is a useful feature. But in my
>>>>>> case, I
>>>>>> > would never have two input files with different schemas. The input
>>>>>> will
>>>>>> > always have only one of the schemas, but I want my new script (with
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*
+
Bill Graham 2012-03-30, 04:59
-
Re: Working with changing schemas (avro) in PigAlex Rovner 2012-04-01, 16:32
Anyone have any experience with elephantbird? Seems like it can handle these cases with ease?
Sent from my iPhone
On Mar 30, 2012, at 12:59 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
> In the TestAvroStorage.testRecordWithFieldSchemaFromTextWithSchemaFile
> there's an example:
>
> STORE avro2 INTO 'output_dir'
> USING org.apache.pig.piggybank.storage.avro.AvroStorage (
> '{"schema_file": "/path/to/schema/file" ,
> "field0": "def:member_id",
> "field1": "def:browser_id",
> "field3": "def:act_content" }'
> );
>
> You specify the file that contains the schema, then you have to map the
> tuple fields to the name of the field in the avro schema. This mapping is a
> drag, but it's currently required.
>
> Note that only the json-style constructor (as opposed to the string array
> appoach) supports schema_file without this uncommitted patch:
> https://issues.apache.org/jira/browse/PIG-2257
>
>
> thanks,
> Bill
>
> On Thu, Mar 29, 2012 at 1:05 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>
>> That's nice! Can you give me an example of how to use it? I am not able to
>> figure it out from the code. The schemaManager is only used at one place
>> after that, and that is when the params contains a "field<number>" key. I
>> don't understand that part. Is there a way I can call it simply like STORE
>> xyz INTO 'abc' USING AvroStorage('schema_file=/path/to/schema/file')?
>>
>>
>>
>> On Wed, Mar 28, 2012 at 5:41 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>>
>>> Yes, the schema can be in HDFS but the documentation for this is lacking.
>>> Search for 'schema_file' here:
>>>
>>>
>>> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
>>>
>>> and here:
>>>
>>>
>>> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
>>>
>>> And be aware of this open JIRA:
>>> https://issues.apache.org/jira/browse/PIG-2257
>>>
>>> And this closed one:
>>> https://issues.apache.org/jira/browse/PIG-2195
>>>
>>> :)
>>>
>>> thanks,
>>> Bill
>>>
>>>
>>> On Wed, Mar 28, 2012 at 5:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>>
>>>> The schema has to be written in the script right? I don't think there is
>>>> any way the schema can be in a file outside the script. That was the
>>>> messyness I was talking about. Or is there a way I can write the schema in
>>>> a separate file? One way I see is to create and store a dummy file with the
>>>> schema
>>>>
>>>>
>>>> Wed, Mar 28, 2012 at 4:19 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> The default value will be part of the new Avro schema definition and
>>>>> Avro should return it to you, so there shouldn't be any code messyness with
>>>>> that approach.
>>>>>
>>>>>
>>>>> On Wed, Mar 28, 2012 at 4:01 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Ok.. you mean I can just use the newer schema to read the old schema
>>>>>> as well, by populating some default value for the missing field. I think
>>>>>> that should work, messy code though!
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]>wrote:
>>>>>>
>>>>>>> If you evolved your schema to just add fields, then you should be
>>>>>>> able to
>>>>>>> use a single schema descriptor file to read both pre- and
>>>>>>> post-evolved data
>>>>>>> objects. This is because one of the rules of new fields in Avro is
>>>>>>> that
>>>>>>> they have to have a default value and be non-null. AvroStorage should
>>>>>>> pick
>>>>>>> that default field up for the old objects. If it doesn't, then that's
>>>>>>> a bug.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> @Bill,
>>>>>>>> I did look at the option of providing input as a parameter while
>>>>>>>> initializing AvroStorage(). But even then, I'll still need to
>>>>>>> change my
>>>>>>>> script to handle the two files because I'll still need to have
Sent from my iPhone
On Mar 30, 2012, at 12:59 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
> In the TestAvroStorage.testRecordWithFieldSchemaFromTextWithSchemaFile
> there's an example:
>
> STORE avro2 INTO 'output_dir'
> USING org.apache.pig.piggybank.storage.avro.AvroStorage (
> '{"schema_file": "/path/to/schema/file" ,
> "field0": "def:member_id",
> "field1": "def:browser_id",
> "field3": "def:act_content" }'
> );
>
> You specify the file that contains the schema, then you have to map the
> tuple fields to the name of the field in the avro schema. This mapping is a
> drag, but it's currently required.
>
> Note that only the json-style constructor (as opposed to the string array
> appoach) supports schema_file without this uncommitted patch:
> https://issues.apache.org/jira/browse/PIG-2257
>
>
> thanks,
> Bill
>
> On Thu, Mar 29, 2012 at 1:05 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>
>> That's nice! Can you give me an example of how to use it? I am not able to
>> figure it out from the code. The schemaManager is only used at one place
>> after that, and that is when the params contains a "field<number>" key. I
>> don't understand that part. Is there a way I can call it simply like STORE
>> xyz INTO 'abc' USING AvroStorage('schema_file=/path/to/schema/file')?
>>
>>
>>
>> On Wed, Mar 28, 2012 at 5:41 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>>
>>> Yes, the schema can be in HDFS but the documentation for this is lacking.
>>> Search for 'schema_file' here:
>>>
>>>
>>> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
>>>
>>> and here:
>>>
>>>
>>> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
>>>
>>> And be aware of this open JIRA:
>>> https://issues.apache.org/jira/browse/PIG-2257
>>>
>>> And this closed one:
>>> https://issues.apache.org/jira/browse/PIG-2195
>>>
>>> :)
>>>
>>> thanks,
>>> Bill
>>>
>>>
>>> On Wed, Mar 28, 2012 at 5:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>>
>>>> The schema has to be written in the script right? I don't think there is
>>>> any way the schema can be in a file outside the script. That was the
>>>> messyness I was talking about. Or is there a way I can write the schema in
>>>> a separate file? One way I see is to create and store a dummy file with the
>>>> schema
>>>>
>>>>
>>>> Wed, Mar 28, 2012 at 4:19 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> The default value will be part of the new Avro schema definition and
>>>>> Avro should return it to you, so there shouldn't be any code messyness with
>>>>> that approach.
>>>>>
>>>>>
>>>>> On Wed, Mar 28, 2012 at 4:01 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Ok.. you mean I can just use the newer schema to read the old schema
>>>>>> as well, by populating some default value for the missing field. I think
>>>>>> that should work, messy code though!
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]>wrote:
>>>>>>
>>>>>>> If you evolved your schema to just add fields, then you should be
>>>>>>> able to
>>>>>>> use a single schema descriptor file to read both pre- and
>>>>>>> post-evolved data
>>>>>>> objects. This is because one of the rules of new fields in Avro is
>>>>>>> that
>>>>>>> they have to have a default value and be non-null. AvroStorage should
>>>>>>> pick
>>>>>>> that default field up for the old objects. If it doesn't, then that's
>>>>>>> a bug.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 28, 2012 at 3:26 PM, IGZ Nick <[EMAIL PROTECTED]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> @Bill,
>>>>>>>> I did look at the option of providing input as a parameter while
>>>>>>>> initializing AvroStorage(). But even then, I'll still need to
>>>>>>> change my
>>>>>>>> script to handle the two files because I'll still need to have
+
Alex Rovner 2012-04-01, 16:32
-
Re: Working with changing schemas (avro) in PigBill Graham 2012-04-02, 05:08
Elephantbird has functionality to integrate with Protobufs and Thrift but
not Avro. When reading and writing messages of either type, EB expects
classes to be generated via schema definitions at build-time. It doesn't
read schemas defs at run-time to dynamically generate messages like one
would do with Avro. Hence EB takes a different approach and doesn't have to
deal with the evolving schema file in the same way as AvroStorage does.
On Sun, Apr 1, 2012 at 9:32 AM, Alex Rovner <[EMAIL PROTECTED]> wrote:
> Anyone have any experience with elephantbird? Seems like it can handle
> these cases with ease?
>
> Sent from my iPhone
>
> On Mar 30, 2012, at 12:59 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
> > In the TestAvroStorage.testRecordWithFieldSchemaFromTextWithSchemaFile
> > there's an example:
> >
> > STORE avro2 INTO 'output_dir'
> > USING org.apache.pig.piggybank.storage.avro.AvroStorage (
> > '{"schema_file": "/path/to/schema/file" ,
> > "field0": "def:member_id",
> > "field1": "def:browser_id",
> > "field3": "def:act_content" }'
> > );
> >
> > You specify the file that contains the schema, then you have to map the
> > tuple fields to the name of the field in the avro schema. This mapping
> is a
> > drag, but it's currently required.
> >
> > Note that only the json-style constructor (as opposed to the string array
> > appoach) supports schema_file without this uncommitted patch:
> > https://issues.apache.org/jira/browse/PIG-2257
> >
> >
> > thanks,
> > Bill
> >
> > On Thu, Mar 29, 2012 at 1:05 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> >
> >> That's nice! Can you give me an example of how to use it? I am not able
> to
> >> figure it out from the code. The schemaManager is only used at one place
> >> after that, and that is when the params contains a "field<number>" key.
> I
> >> don't understand that part. Is there a way I can call it simply like
> STORE
> >> xyz INTO 'abc' USING AvroStorage('schema_file=/path/to/schema/file')?
> >>
> >>
> >>
> >> On Wed, Mar 28, 2012 at 5:41 PM, Bill Graham <[EMAIL PROTECTED]>
> wrote:
> >>
> >>> Yes, the schema can be in HDFS but the documentation for this is
> lacking.
> >>> Search for 'schema_file' here:
> >>>
> >>>
> >>>
> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
> >>>
> >>> and here:
> >>>
> >>>
> >>>
> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
> >>>
> >>> And be aware of this open JIRA:
> >>> https://issues.apache.org/jira/browse/PIG-2257
> >>>
> >>> And this closed one:
> >>> https://issues.apache.org/jira/browse/PIG-2195
> >>>
> >>> :)
> >>>
> >>> thanks,
> >>> Bill
> >>>
> >>>
> >>> On Wed, Mar 28, 2012 at 5:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> >>>
> >>>> The schema has to be written in the script right? I don't think there
> is
> >>>> any way the schema can be in a file outside the script. That was the
> >>>> messyness I was talking about. Or is there a way I can write the
> schema in
> >>>> a separate file? One way I see is to create and store a dummy file
> with the
> >>>> schema
> >>>>
> >>>>
> >>>> Wed, Mar 28, 2012 at 4:19 PM, Bill Graham <[EMAIL PROTECTED]>
> wrote:
> >>>>
> >>>>> The default value will be part of the new Avro schema definition and
> >>>>> Avro should return it to you, so there shouldn't be any code
> messyness with
> >>>>> that approach.
> >>>>>
> >>>>>
> >>>>> On Wed, Mar 28, 2012 at 4:01 PM, IGZ Nick <[EMAIL PROTECTED]>
> wrote:
> >>>>>
> >>>>>> Ok.. you mean I can just use the newer schema to read the old schema
> >>>>>> as well, by populating some default value for the missing field. I
> think
> >>>>>> that should work, messy code though!
> >>>>>>
> >>>>>> Thanks!
> >>>>>>
> >>>>>> On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]
> >wrote:
> >>>>>>
> >>>>>>> If you evolved your schema to just add fields, then you should be
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*
not Avro. When reading and writing messages of either type, EB expects
classes to be generated via schema definitions at build-time. It doesn't
read schemas defs at run-time to dynamically generate messages like one
would do with Avro. Hence EB takes a different approach and doesn't have to
deal with the evolving schema file in the same way as AvroStorage does.
On Sun, Apr 1, 2012 at 9:32 AM, Alex Rovner <[EMAIL PROTECTED]> wrote:
> Anyone have any experience with elephantbird? Seems like it can handle
> these cases with ease?
>
> Sent from my iPhone
>
> On Mar 30, 2012, at 12:59 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
> > In the TestAvroStorage.testRecordWithFieldSchemaFromTextWithSchemaFile
> > there's an example:
> >
> > STORE avro2 INTO 'output_dir'
> > USING org.apache.pig.piggybank.storage.avro.AvroStorage (
> > '{"schema_file": "/path/to/schema/file" ,
> > "field0": "def:member_id",
> > "field1": "def:browser_id",
> > "field3": "def:act_content" }'
> > );
> >
> > You specify the file that contains the schema, then you have to map the
> > tuple fields to the name of the field in the avro schema. This mapping
> is a
> > drag, but it's currently required.
> >
> > Note that only the json-style constructor (as opposed to the string array
> > appoach) supports schema_file without this uncommitted patch:
> > https://issues.apache.org/jira/browse/PIG-2257
> >
> >
> > thanks,
> > Bill
> >
> > On Thu, Mar 29, 2012 at 1:05 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> >
> >> That's nice! Can you give me an example of how to use it? I am not able
> to
> >> figure it out from the code. The schemaManager is only used at one place
> >> after that, and that is when the params contains a "field<number>" key.
> I
> >> don't understand that part. Is there a way I can call it simply like
> STORE
> >> xyz INTO 'abc' USING AvroStorage('schema_file=/path/to/schema/file')?
> >>
> >>
> >>
> >> On Wed, Mar 28, 2012 at 5:41 PM, Bill Graham <[EMAIL PROTECTED]>
> wrote:
> >>
> >>> Yes, the schema can be in HDFS but the documentation for this is
> lacking.
> >>> Search for 'schema_file' here:
> >>>
> >>>
> >>>
> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
> >>>
> >>> and here:
> >>>
> >>>
> >>>
> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
> >>>
> >>> And be aware of this open JIRA:
> >>> https://issues.apache.org/jira/browse/PIG-2257
> >>>
> >>> And this closed one:
> >>> https://issues.apache.org/jira/browse/PIG-2195
> >>>
> >>> :)
> >>>
> >>> thanks,
> >>> Bill
> >>>
> >>>
> >>> On Wed, Mar 28, 2012 at 5:26 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> >>>
> >>>> The schema has to be written in the script right? I don't think there
> is
> >>>> any way the schema can be in a file outside the script. That was the
> >>>> messyness I was talking about. Or is there a way I can write the
> schema in
> >>>> a separate file? One way I see is to create and store a dummy file
> with the
> >>>> schema
> >>>>
> >>>>
> >>>> Wed, Mar 28, 2012 at 4:19 PM, Bill Graham <[EMAIL PROTECTED]>
> wrote:
> >>>>
> >>>>> The default value will be part of the new Avro schema definition and
> >>>>> Avro should return it to you, so there shouldn't be any code
> messyness with
> >>>>> that approach.
> >>>>>
> >>>>>
> >>>>> On Wed, Mar 28, 2012 at 4:01 PM, IGZ Nick <[EMAIL PROTECTED]>
> wrote:
> >>>>>
> >>>>>> Ok.. you mean I can just use the newer schema to read the old schema
> >>>>>> as well, by populating some default value for the missing field. I
> think
> >>>>>> that should work, messy code though!
> >>>>>>
> >>>>>> Thanks!
> >>>>>>
> >>>>>> On Wed, Mar 28, 2012 at 3:53 PM, Bill Graham <[EMAIL PROTECTED]
> >wrote:
> >>>>>>
> >>>>>>> If you evolved your schema to just add fields, then you should be
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*
+
Bill Graham 2012-04-02, 05:08
-
Re: Working with changing schemas (avro) in PigAlex Rovner 2012-04-02, 17:18
Bill,
It would seem that you will hit the same issues though. Image you are
processing log files from an application. As your schema changes, you
certainly do not want to reprocess all the historic logs. I believe
Protobufs and Thrift handle these cases gracefully by inserting nulls into
expected columns that are not found?
Alex
On Mon, Apr 2, 2012 at 1:08 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
> Elephantbird has functionality to integrate with Protobufs and Thrift but
> not Avro. When reading and writing messages of either type, EB expects
> classes to be generated via schema definitions at build-time. It doesn't
> read schemas defs at run-time to dynamically generate messages like one
> would do with Avro. Hence EB takes a different approach and doesn't have to
> deal with the evolving schema file in the same way as AvroStorage does.
>
>
> On Sun, Apr 1, 2012 at 9:32 AM, Alex Rovner <[EMAIL PROTECTED]> wrote:
>
> > Anyone have any experience with elephantbird? Seems like it can handle
> > these cases with ease?
> >
> > Sent from my iPhone
> >
> > On Mar 30, 2012, at 12:59 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
> >
> > > In the TestAvroStorage.testRecordWithFieldSchemaFromTextWithSchemaFile
> > > there's an example:
> > >
> > > STORE avro2 INTO 'output_dir'
> > > USING org.apache.pig.piggybank.storage.avro.AvroStorage (
> > > '{"schema_file": "/path/to/schema/file" ,
> > > "field0": "def:member_id",
> > > "field1": "def:browser_id",
> > > "field3": "def:act_content" }'
> > > );
> > >
> > > You specify the file that contains the schema, then you have to map the
> > > tuple fields to the name of the field in the avro schema. This mapping
> > is a
> > > drag, but it's currently required.
> > >
> > > Note that only the json-style constructor (as opposed to the string
> array
> > > appoach) supports schema_file without this uncommitted patch:
> > > https://issues.apache.org/jira/browse/PIG-2257
> > >
> > >
> > > thanks,
> > > Bill
> > >
> > > On Thu, Mar 29, 2012 at 1:05 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> > >
> > >> That's nice! Can you give me an example of how to use it? I am not
> able
> > to
> > >> figure it out from the code. The schemaManager is only used at one
> place
> > >> after that, and that is when the params contains a "field<number>"
> key.
> > I
> > >> don't understand that part. Is there a way I can call it simply like
> > STORE
> > >> xyz INTO 'abc' USING AvroStorage('schema_file=/path/to/schema/file')?
> > >>
> > >>
> > >>
> > >> On Wed, Mar 28, 2012 at 5:41 PM, Bill Graham <[EMAIL PROTECTED]>
> > wrote:
> > >>
> > >>> Yes, the schema can be in HDFS but the documentation for this is
> > lacking.
> > >>> Search for 'schema_file' here:
> > >>>
> > >>>
> > >>>
> >
> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
> > >>>
> > >>> and here:
> > >>>
> > >>>
> > >>>
> >
> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
> > >>>
> > >>> And be aware of this open JIRA:
> > >>> https://issues.apache.org/jira/browse/PIG-2257
> > >>>
> > >>> And this closed one:
> > >>> https://issues.apache.org/jira/browse/PIG-2195
> > >>>
> > >>> :)
> > >>>
> > >>> thanks,
> > >>> Bill
> > >>>
> > >>>
> > >>> On Wed, Mar 28, 2012 at 5:26 PM, IGZ Nick <[EMAIL PROTECTED]>
> wrote:
> > >>>
> > >>>> The schema has to be written in the script right? I don't think
> there
> > is
> > >>>> any way the schema can be in a file outside the script. That was the
> > >>>> messyness I was talking about. Or is there a way I can write the
> > schema in
> > >>>> a separate file? One way I see is to create and store a dummy file
> > with the
> > >>>> schema
> > >>>>
> > >>>>
> > >>>> Wed, Mar 28, 2012 at 4:19 PM, Bill Graham <[EMAIL PROTECTED]>
> > wrote:
> > >>>>
> > >>>>> The default value will be part of the new Avro schema definition
> and
It would seem that you will hit the same issues though. Image you are
processing log files from an application. As your schema changes, you
certainly do not want to reprocess all the historic logs. I believe
Protobufs and Thrift handle these cases gracefully by inserting nulls into
expected columns that are not found?
Alex
On Mon, Apr 2, 2012 at 1:08 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
> Elephantbird has functionality to integrate with Protobufs and Thrift but
> not Avro. When reading and writing messages of either type, EB expects
> classes to be generated via schema definitions at build-time. It doesn't
> read schemas defs at run-time to dynamically generate messages like one
> would do with Avro. Hence EB takes a different approach and doesn't have to
> deal with the evolving schema file in the same way as AvroStorage does.
>
>
> On Sun, Apr 1, 2012 at 9:32 AM, Alex Rovner <[EMAIL PROTECTED]> wrote:
>
> > Anyone have any experience with elephantbird? Seems like it can handle
> > these cases with ease?
> >
> > Sent from my iPhone
> >
> > On Mar 30, 2012, at 12:59 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
> >
> > > In the TestAvroStorage.testRecordWithFieldSchemaFromTextWithSchemaFile
> > > there's an example:
> > >
> > > STORE avro2 INTO 'output_dir'
> > > USING org.apache.pig.piggybank.storage.avro.AvroStorage (
> > > '{"schema_file": "/path/to/schema/file" ,
> > > "field0": "def:member_id",
> > > "field1": "def:browser_id",
> > > "field3": "def:act_content" }'
> > > );
> > >
> > > You specify the file that contains the schema, then you have to map the
> > > tuple fields to the name of the field in the avro schema. This mapping
> > is a
> > > drag, but it's currently required.
> > >
> > > Note that only the json-style constructor (as opposed to the string
> array
> > > appoach) supports schema_file without this uncommitted patch:
> > > https://issues.apache.org/jira/browse/PIG-2257
> > >
> > >
> > > thanks,
> > > Bill
> > >
> > > On Thu, Mar 29, 2012 at 1:05 PM, IGZ Nick <[EMAIL PROTECTED]> wrote:
> > >
> > >> That's nice! Can you give me an example of how to use it? I am not
> able
> > to
> > >> figure it out from the code. The schemaManager is only used at one
> place
> > >> after that, and that is when the params contains a "field<number>"
> key.
> > I
> > >> don't understand that part. Is there a way I can call it simply like
> > STORE
> > >> xyz INTO 'abc' USING AvroStorage('schema_file=/path/to/schema/file')?
> > >>
> > >>
> > >>
> > >> On Wed, Mar 28, 2012 at 5:41 PM, Bill Graham <[EMAIL PROTECTED]>
> > wrote:
> > >>
> > >>> Yes, the schema can be in HDFS but the documentation for this is
> > lacking.
> > >>> Search for 'schema_file' here:
> > >>>
> > >>>
> > >>>
> >
> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
> > >>>
> > >>> and here:
> > >>>
> > >>>
> > >>>
> >
> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
> > >>>
> > >>> And be aware of this open JIRA:
> > >>> https://issues.apache.org/jira/browse/PIG-2257
> > >>>
> > >>> And this closed one:
> > >>> https://issues.apache.org/jira/browse/PIG-2195
> > >>>
> > >>> :)
> > >>>
> > >>> thanks,
> > >>> Bill
> > >>>
> > >>>
> > >>> On Wed, Mar 28, 2012 at 5:26 PM, IGZ Nick <[EMAIL PROTECTED]>
> wrote:
> > >>>
> > >>>> The schema has to be written in the script right? I don't think
> there
> > is
> > >>>> any way the schema can be in a file outside the script. That was the
> > >>>> messyness I was talking about. Or is there a way I can write the
> > schema in
> > >>>> a separate file? One way I see is to create and store a dummy file
> > with the
> > >>>> schema
> > >>>>
> > >>>>
> > >>>> Wed, Mar 28, 2012 at 4:19 PM, Bill Graham <[EMAIL PROTECTED]>
> > wrote:
> > >>>>
> > >>>>> The default value will be part of the new Avro schema definition
> and
+
Alex Rovner 2012-04-02, 17:18
-
Re: Working with changing schemas (avro) in PigBill Graham 2012-04-02, 17:33
Yes and Avro has similar functionality to return a default value when the
schema evolves.
Re your comment:
Seems like it can handle these cases with ease?
I thought the "these cases" you were referring to was the issue referencing
a schema definition file at runtime, which is AvroStorage does. Thrift and
Protobufs via EB don't have this issue, since they deal with it at compile
time with codegen.
On Mon, Apr 2, 2012 at 10:18 AM, Alex Rovner <[EMAIL PROTECTED]> wrote:
> Bill,
>
> It would seem that you will hit the same issues though. Image you are
> processing log files from an application. As your schema changes, you
> certainly do not want to reprocess all the historic logs. I believe
> Protobufs and Thrift handle these cases gracefully by inserting nulls into
> expected columns that are not found?
>
> Alex
>
>
> On Mon, Apr 2, 2012 at 1:08 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
>> Elephantbird has functionality to integrate with Protobufs and Thrift but
>> not Avro. When reading and writing messages of either type, EB expects
>> classes to be generated via schema definitions at build-time. It doesn't
>> read schemas defs at run-time to dynamically generate messages like one
>> would do with Avro. Hence EB takes a different approach and doesn't have
>> to
>> deal with the evolving schema file in the same way as AvroStorage does.
>>
>>
>> On Sun, Apr 1, 2012 at 9:32 AM, Alex Rovner <[EMAIL PROTECTED]> wrote:
>>
>> > Anyone have any experience with elephantbird? Seems like it can handle
>> > these cases with ease?
>> >
>> > Sent from my iPhone
>> >
>> > On Mar 30, 2012, at 12:59 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
>> >
>> > > In the TestAvroStorage.testRecordWithFieldSchemaFromTextWithSchemaFile
>> > > there's an example:
>> > >
>> > > STORE avro2 INTO 'output_dir'
>> > > USING org.apache.pig.piggybank.storage.avro.AvroStorage (
>> > > '{"schema_file": "/path/to/schema/file" ,
>> > > "field0": "def:member_id",
>> > > "field1": "def:browser_id",
>> > > "field3": "def:act_content" }'
>> > > );
>> > >
>> > > You specify the file that contains the schema, then you have to map
>> the
>> > > tuple fields to the name of the field in the avro schema. This mapping
>> > is a
>> > > drag, but it's currently required.
>> > >
>> > > Note that only the json-style constructor (as opposed to the string
>> array
>> > > appoach) supports schema_file without this uncommitted patch:
>> > > https://issues.apache.org/jira/browse/PIG-2257
>> > >
>> > >
>> > > thanks,
>> > > Bill
>> > >
>> > > On Thu, Mar 29, 2012 at 1:05 PM, IGZ Nick <[EMAIL PROTECTED]>
>> wrote:
>> > >
>> > >> That's nice! Can you give me an example of how to use it? I am not
>> able
>> > to
>> > >> figure it out from the code. The schemaManager is only used at one
>> place
>> > >> after that, and that is when the params contains a "field<number>"
>> key.
>> > I
>> > >> don't understand that part. Is there a way I can call it simply like
>> > STORE
>> > >> xyz INTO 'abc' USING AvroStorage('schema_file=/path/to/schema/file')?
>> > >>
>> > >>
>> > >>
>> > >> On Wed, Mar 28, 2012 at 5:41 PM, Bill Graham <[EMAIL PROTECTED]>
>> > wrote:
>> > >>
>> > >>> Yes, the schema can be in HDFS but the documentation for this is
>> > lacking.
>> > >>> Search for 'schema_file' here:
>> > >>>
>> > >>>
>> > >>>
>> >
>> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
>> > >>>
>> > >>> and here:
>> > >>>
>> > >>>
>> > >>>
>> >
>> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
>> > >>>
>> > >>> And be aware of this open JIRA:
>> > >>> https://issues.apache.org/jira/browse/PIG-2257
>> > >>>
>> > >>> And this closed one:
>> > >>> https://issues.apache.org/jira/browse/PIG-2195
>> > >>>
>> > >>> :)
>> > >>>
>> > >>> thanks,
>> > >>> Bill
>> > >>>
>> > >>>
>> > >>> On Wed, Mar 28, 2012 at 5:26 PM, IGZ Nick <[EMAIL PROTECTED]>
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*
schema evolves.
Re your comment:
Seems like it can handle these cases with ease?
I thought the "these cases" you were referring to was the issue referencing
a schema definition file at runtime, which is AvroStorage does. Thrift and
Protobufs via EB don't have this issue, since they deal with it at compile
time with codegen.
On Mon, Apr 2, 2012 at 10:18 AM, Alex Rovner <[EMAIL PROTECTED]> wrote:
> Bill,
>
> It would seem that you will hit the same issues though. Image you are
> processing log files from an application. As your schema changes, you
> certainly do not want to reprocess all the historic logs. I believe
> Protobufs and Thrift handle these cases gracefully by inserting nulls into
> expected columns that are not found?
>
> Alex
>
>
> On Mon, Apr 2, 2012 at 1:08 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
>> Elephantbird has functionality to integrate with Protobufs and Thrift but
>> not Avro. When reading and writing messages of either type, EB expects
>> classes to be generated via schema definitions at build-time. It doesn't
>> read schemas defs at run-time to dynamically generate messages like one
>> would do with Avro. Hence EB takes a different approach and doesn't have
>> to
>> deal with the evolving schema file in the same way as AvroStorage does.
>>
>>
>> On Sun, Apr 1, 2012 at 9:32 AM, Alex Rovner <[EMAIL PROTECTED]> wrote:
>>
>> > Anyone have any experience with elephantbird? Seems like it can handle
>> > these cases with ease?
>> >
>> > Sent from my iPhone
>> >
>> > On Mar 30, 2012, at 12:59 AM, Bill Graham <[EMAIL PROTECTED]> wrote:
>> >
>> > > In the TestAvroStorage.testRecordWithFieldSchemaFromTextWithSchemaFile
>> > > there's an example:
>> > >
>> > > STORE avro2 INTO 'output_dir'
>> > > USING org.apache.pig.piggybank.storage.avro.AvroStorage (
>> > > '{"schema_file": "/path/to/schema/file" ,
>> > > "field0": "def:member_id",
>> > > "field1": "def:browser_id",
>> > > "field3": "def:act_content" }'
>> > > );
>> > >
>> > > You specify the file that contains the schema, then you have to map
>> the
>> > > tuple fields to the name of the field in the avro schema. This mapping
>> > is a
>> > > drag, but it's currently required.
>> > >
>> > > Note that only the json-style constructor (as opposed to the string
>> array
>> > > appoach) supports schema_file without this uncommitted patch:
>> > > https://issues.apache.org/jira/browse/PIG-2257
>> > >
>> > >
>> > > thanks,
>> > > Bill
>> > >
>> > > On Thu, Mar 29, 2012 at 1:05 PM, IGZ Nick <[EMAIL PROTECTED]>
>> wrote:
>> > >
>> > >> That's nice! Can you give me an example of how to use it? I am not
>> able
>> > to
>> > >> figure it out from the code. The schemaManager is only used at one
>> place
>> > >> after that, and that is when the params contains a "field<number>"
>> key.
>> > I
>> > >> don't understand that part. Is there a way I can call it simply like
>> > STORE
>> > >> xyz INTO 'abc' USING AvroStorage('schema_file=/path/to/schema/file')?
>> > >>
>> > >>
>> > >>
>> > >> On Wed, Mar 28, 2012 at 5:41 PM, Bill Graham <[EMAIL PROTECTED]>
>> > wrote:
>> > >>
>> > >>> Yes, the schema can be in HDFS but the documentation for this is
>> > lacking.
>> > >>> Search for 'schema_file' here:
>> > >>>
>> > >>>
>> > >>>
>> >
>> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java
>> > >>>
>> > >>> and here:
>> > >>>
>> > >>>
>> > >>>
>> >
>> http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
>> > >>>
>> > >>> And be aware of this open JIRA:
>> > >>> https://issues.apache.org/jira/browse/PIG-2257
>> > >>>
>> > >>> And this closed one:
>> > >>> https://issues.apache.org/jira/browse/PIG-2195
>> > >>>
>> > >>> :)
>> > >>>
>> > >>> thanks,
>> > >>> Bill
>> > >>>
>> > >>>
>> > >>> On Wed, Mar 28, 2012 at 5:26 PM, IGZ Nick <[EMAIL PROTECTED]>
*Note that I'm no longer using my Yahoo! email address. Please email me at
[EMAIL PROTECTED] going forward.*
+
Bill Graham 2012-04-02, 17:33
