Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - How to Use AvroStorage to read avro data and convert into a new (compatible) schema


Copy link to this message
-
Re: How to Use AvroStorage to read avro data and convert into a new (compatible) schema
Russell Jurney 2012-10-06, 03:40
Try loading the data in schema one, and storing it/transforming it in
schema two in a new location?

Then try to load it, and see what it is. This may not be feasible if
your data is large, but worth a try to help debug?

Russell Jurney http://datasyndrome.com

On Oct 5, 2012, at 7:48 PM, Runping Qi <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I have the following problem to solve:
>
> I have data in HDFS written out using AvroStorage with schema1.
> I have schema2 which is different than schema1 but they are compatible.
> Now, I want to use AvroStorage to load the data and convert to schema2.
> How can I achieve that?
>
> I tried something like:
>
>    data = load 'my_data_dir' using AvroStorage('schema', SCHEMA2_STRING);
>    describe data;
>
> The schema for the loaded-in data is still in schema1, not in schema2.
>
> Any help is highly appreciated.
>
> Thanks,
>
> Runping Qi
> A9.COM