Hi All,

I am using Avro for processing/encoding billions of rows using Hadoop.  When
I attempted to upgrade to Avro 1.5 I got an error about illegal field names.
I notice this case happens when I use a O-¹ character or a O.¹ in a field
name.  

Are there any other restrictions on fields names I need to know when
upgrading?

Thanks,
Matt

iCrossing Privileged and Confidential Information
This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information of iCrossing. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

> I am using Avro for processing/encoding billions of rows using Hadoop.  When I attempted to upgrade to Avro 1.5 I got an error about illegal field names.  I notice this case happens when I use a ‘-’ character or a ‘.’ in a field name.  
>
> Are there any other restrictions on fields names I need to know when upgrading?

You can find the full details in the Avro spec [1].  It's basically just like the identifier restrictions in most programming languages.  Names can only contain letters, numbers, and underscores, and can't start with a number.  Dots are used to created namespaces, just like fully-qualified Java identifiers.

[1] http://avro.apache.org/docs/current/spec.html

cheers
–doug

OK, thanks.  This is a change from Avro 1.4 which will make older Avro files
unreadable by 1.5 until the field names are changed.

Can we get away with "tweeking" the field name in the JSON meta data portion
of the file, or do we have to re-write the whole file again with the new
meta data?

On 6/10/11 12:23 PM, "Douglas Creager" <[EMAIL PROTECTED]> wrote:

>> I am using Avro for processing/encoding billions of rows using Hadoop.  When
>> I attempted to upgrade to Avro 1.5 I got an error about illegal field names.
>> I notice this case happens when I use a O-¹ character or a O.¹ in a field
>> name.  
>>
>> Are there any other restrictions on fields names I need to know when
>> upgrading?
>
> You can find the full details in the Avro spec [1].  It's basically just like
> the identifier restrictions in most programming languages.  Names can only
> contain letters, numbers, and underscores, and can't start with a number.
> Dots are used to created namespaces, just like fully-qualified Java
> identifiers.
>
> [1] http://avro.apache.org/docs/current/spec.html
>
> cheers
> ­doug
iCrossing Privileged and Confidential Information
This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information of iCrossing. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

I'd run into a similar issue, where I used a record name with a "-" in it when creating files with Avro 1.4.

So I've got files that are valid/readable with 1.4, but seem to be unusable with 1.5.

-- Ken

On Jun 10, 2011, at 12:32pm, Matt Pouttu-Clarke wrote:

> OK, thanks.  This is a change from Avro 1.4 which will make older Avro files
> unreadable by 1.5 until the field names are changed.
>
> Can we get away with "tweeking" the field name in the JSON meta data portion
> of the file, or do we have to re-write the whole file again with the new
> meta data?
>
> On 6/10/11 12:23 PM, "Douglas Creager" <[EMAIL PROTECTED]> wrote:
>
>>> I am using Avro for processing/encoding billions of rows using Hadoop.  When
>>> I attempted to upgrade to Avro 1.5 I got an error about illegal field names.
>>> I notice this case happens when I use a O-’ character or a O.’ in a field
>>> name.  
>>>
>>> Are there any other restrictions on fields names I need to know when
>>> upgrading?
>>
>> You can find the full details in the Avro spec [1].  It's basically just like
>> the identifier restrictions in most programming languages.  Names can only
>> contain letters, numbers, and underscores, and can't start with a number.
>> Dots are used to created namespaces, just like fully-qualified Java
>> identifiers.
>>
>> [1] http://avro.apache.org/docs/current/spec.html
>>
>> cheers
>> –doug
>
>
> iCrossing Privileged and Confidential Information
> This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information of iCrossing. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
>
>

--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
custom data mining solutions

Ok, there was a bug in 1.4 that allowed for invalid names.  Sounds like we
need something that allows reading some older files.
Please open a JIRA ticket explaining the issue, we can discuss
solutions/workarounds there.

On 6/10/11 1:48 PM, "Ken Krugler" <[EMAIL PROTECTED]> wrote:

>I'd run into a similar issue, where I used a record name with a "-" in it
>when creating files with Avro 1.4.
>
>So I've got files that are valid/readable with 1.4, but seem to be
>unusable with 1.5.
>
>-- Ken
>
>On Jun 10, 2011, at 12:32pm, Matt Pouttu-Clarke wrote:
>
>> OK, thanks.  This is a change from Avro 1.4 which will make older Avro
>>files
>> unreadable by 1.5 until the field names are changed.
>>
>> Can we get away with "tweeking" the field name in the JSON meta data
>>portion
>> of the file, or do we have to re-write the whole file again with the new
>> meta data?
>>
>> On 6/10/11 12:23 PM, "Douglas Creager" <[EMAIL PROTECTED]> wrote:
>>
>>>> I am using Avro for processing/encoding billions of rows using
>>>>Hadoop.  When
>>>> I attempted to upgrade to Avro 1.5 I got an error about illegal field
>>>>names.
>>>> I notice this case happens when I use a O-¹ character or a O.¹ in a
>>>>field
>>>> name.  
>>>>
>>>> Are there any other restrictions on fields names I need to know when
>>>> upgrading?
>>>
>>> You can find the full details in the Avro spec [1].  It's basically
>>>just like
>>> the identifier restrictions in most programming languages.  Names can
>>>only
>>> contain letters, numbers, and underscores, and can't start with a
>>>number.
>>> Dots are used to created namespaces, just like fully-qualified Java
>>> identifiers.
>>>
>>> [1] http://avro.apache.org/docs/current/spec.html
>>>
>>> cheers
>>> ­doug
>>
>>
>> iCrossing Privileged and Confidential Information
>> This email message is for the sole use of the intended recipient(s) and
>>may contain confidential and privileged information of iCrossing. Any
>>unauthorized review, use, disclosure or distribution is prohibited. If
>>you are not the intended recipient, please contact the sender by reply
>>email and destroy all copies of the original message.
>>
>>
>
>--------------------------
>Ken Krugler
>+1 530-210-6378
>http://bixolabs.com
>custom data mining solutions
>
>
>
>
>
>

On Jun 10, 2011, at 4:01pm, Scott Carey wrote:

> Ok, there was a bug in 1.4 that allowed for invalid names.  Sounds like we
> need something that allows reading some older files.
> Please open a JIRA ticket explaining the issue, we can discuss
> solutions/workarounds there.

https://issues.apache.org/jira/browse/AVRO-838

Thanks,

-- Ken
>
> On 6/10/11 1:48 PM, "Ken Krugler" <[EMAIL PROTECTED]> wrote:
>
>> I'd run into a similar issue, where I used a record name with a "-" in it
>> when creating files with Avro 1.4.
>>
>> So I've got files that are valid/readable with 1.4, but seem to be
>> unusable with 1.5.
>>
>> -- Ken
>>
>> On Jun 10, 2011, at 12:32pm, Matt Pouttu-Clarke wrote:
>>
>>> OK, thanks.  This is a change from Avro 1.4 which will make older Avro
>>> files
>>> unreadable by 1.5 until the field names are changed.
>>>
>>> Can we get away with "tweeking" the field name in the JSON meta data
>>> portion
>>> of the file, or do we have to re-write the whole file again with the new
>>> meta data?
>>>
>>> On 6/10/11 12:23 PM, "Douglas Creager" <[EMAIL PROTECTED]> wrote:
>>>
>>>>> I am using Avro for processing/encoding billions of rows using
>>>>> Hadoop.  When
>>>>> I attempted to upgrade to Avro 1.5 I got an error about illegal field
>>>>> names.
>>>>> I notice this case happens when I use a O-¹ character or a O.¹ in a
>>>>> field
>>>>> name.  
>>>>>
>>>>> Are there any other restrictions on fields names I need to know when
>>>>> upgrading?
>>>>
>>>> You can find the full details in the Avro spec [1].  It's basically
>>>> just like
>>>> the identifier restrictions in most programming languages.  Names can
>>>> only
>>>> contain letters, numbers, and underscores, and can't start with a
>>>> number.
>>>> Dots are used to created namespaces, just like fully-qualified Java
>>>> identifiers.
>>>>
>>>> [1] http://avro.apache.org/docs/current/spec.html
>>>>
>>>> cheers
>>>> doug
>>>
>>>
>>> iCrossing Privileged and Confidential Information
>>> This email message is for the sole use of the intended recipient(s) and
>>> may contain confidential and privileged information of iCrossing. Any
>>> unauthorized review, use, disclosure or distribution is prohibited. If
>>> you are not the intended recipient, please contact the sender by reply
>>> email and destroy all copies of the original message.
>>>
>>>
>>
>> --------------------------
>> Ken Krugler
>> +1 530-210-6378
>> http://bixolabs.com
>> custom data mining solutions
>>
>>
>>
>>
>>
>>
>

--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
custom data mining solutions

On 06/10/2011 09:32 PM, Matt Pouttu-Clarke wrote:
> Can we get away with "tweeking" the field name in the JSON meta data portion
> of the file, or do we have to re-write the whole file again with the new
> meta data?

Yes, you should be able to change that one character in the binary file
and then read the file.  But a better compatibility fix would be good.

Doug