Hi all,

I'm using pig with protobuf and I have some byte fields containing serialized protobuf data.  Is it possible to handle this nested serialized data with pig?

ex.
message A {
  required bytes data = 1   // serialized message B
}
message B {
  ...
}

Thanks,
Ben

Ben,

take a look at elephantbird https://github.com/kevinweil/elephant-bird

Jon

2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>

> Hi all,
>
> I'm using pig with protobuf and I have some byte fields containing
> serialized protobuf data.  Is it possible to handle this nested serialized
> data with pig?
>
> ex.
> message A {
>  required bytes data = 1   // serialized message B
> }
> message B {
>  ...
> }
>
> Thanks,
> Ben

Thanks Jon,

I'm using elephant-bird, and it allows me to load message A.  I'm not sure if it has pig bindings allowing me to deserialize message B from message A though.

Thanks,
Ben

On Mar 26, 2012, at 4:09 PM, Jonathan Coveney wrote:

> Ben,
>
> take a look at elephantbird https://github.com/kevinweil/elephant-bird
>
> Jon
>
> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
>
>> Hi all,
>>
>> I'm using pig with protobuf and I have some byte fields containing
>> serialized protobuf data.  Is it possible to handle this nested serialized
>> data with pig?
>>
>> ex.
>> message A {
>> required bytes data = 1   // serialized message B
>> }
>> message B {
>> ...
>> }
>>
>> Thanks,
>> Ben

Look up the util class ProtobuftoPig

2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>

> Thanks Jon,
>
> I'm using elephant-bird, and it allows me to load message A.  I'm not sure
> if it has pig bindings allowing me to deserialize message B from message A
> though.
>
> Thanks,
> Ben
>
> On Mar 26, 2012, at 4:09 PM, Jonathan Coveney wrote:
>
> > Ben,
> >
> > take a look at elephantbird https://github.com/kevinweil/elephant-bird
> >
> > Jon
> >
> > 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
> >
> >> Hi all,
> >>
> >> I'm using pig with protobuf and I have some byte fields containing
> >> serialized protobuf data.  Is it possible to handle this nested
> serialized
> >> data with pig?
> >>
> >> ex.
> >> message A {
> >> required bytes data = 1   // serialized message B
> >> }
> >> message B {
> >> ...
> >> }
> >>
> >> Thanks,
> >> Ben
>
>

Hey Jon,

I'm not seeing anything in ProtobuftoPig that is specific to handling protobuf byte fields containing serialized protobuf.

Do you have a Pig example?

Thanks,
Ben

On Mar 26, 2012, at 4:14 PM, Jonathan Coveney wrote:

> Look up the util class ProtobuftoPig
>
> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
>
>> Thanks Jon,
>>
>> I'm using elephant-bird, and it allows me to load message A.  I'm not sure
>> if it has pig bindings allowing me to deserialize message B from message A
>> though.
>>
>> Thanks,
>> Ben
>>
>> On Mar 26, 2012, at 4:09 PM, Jonathan Coveney wrote:
>>
>>> Ben,
>>>
>>> take a look at elephantbird https://github.com/kevinweil/elephant-bird
>>>
>>> Jon
>>>
>>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
>>>
>>>> Hi all,
>>>>
>>>> I'm using pig with protobuf and I have some byte fields containing
>>>> serialized protobuf data.  Is it possible to handle this nested
>> serialized
>>>> data with pig?
>>>>
>>>> ex.
>>>> message A {
>>>> required bytes data = 1   // serialized message B
>>>> }
>>>> message B {
>>>> ...
>>>> }
>>>>
>>>> Thanks,
>>>> Ben
>>
>>

I think you want ProtobufBytesToTuple
(https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/piggybank/ProtobufBytesToTuple.java)

This uses ProtobufToPig to do the actual conversion, and wraps it in a UDF.

D

On Tue, Mar 27, 2012 at 7:21 PM, Benjamin Juhn <[EMAIL PROTECTED]> wrote:
> Hey Jon,
>
> I'm not seeing anything in ProtobuftoPig that is specific to handling protobuf byte fields containing serialized protobuf.
>
> Do you have a Pig example?
>
> Thanks,
> Ben
>
> On Mar 26, 2012, at 4:14 PM, Jonathan Coveney wrote:
>
>> Look up the util class ProtobuftoPig
>>
>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
>>
>>> Thanks Jon,
>>>
>>> I'm using elephant-bird, and it allows me to load message A.  I'm not sure
>>> if it has pig bindings allowing me to deserialize message B from message A
>>> though.
>>>
>>> Thanks,
>>> Ben
>>>
>>> On Mar 26, 2012, at 4:09 PM, Jonathan Coveney wrote:
>>>
>>>> Ben,
>>>>
>>>> take a look at elephantbird https://github.com/kevinweil/elephant-bird
>>>>
>>>> Jon
>>>>
>>>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
>>>>
>>>>> Hi all,
>>>>>
>>>>> I'm using pig with protobuf and I have some byte fields containing
>>>>> serialized protobuf data.  Is it possible to handle this nested
>>> serialized
>>>>> data with pig?
>>>>>
>>>>> ex.
>>>>> message A {
>>>>> required bytes data = 1   // serialized message B
>>>>> }
>>>>> message B {
>>>>> ...
>>>>> }
>>>>>
>>>>> Thanks,
>>>>> Ben
>>>
>>>
>

Thanks Dmitriy.  Doesn't look like that class supports extensions.  Am I missing something?

- Ben

On Mar 27, 2012, at 10:01 PM, Dmitriy Ryaboy wrote:

> I think you want ProtobufBytesToTuple
> (https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/piggybank/ProtobufBytesToTuple.java)
>
> This uses ProtobufToPig to do the actual conversion, and wraps it in a UDF.
>
> D
>
> On Tue, Mar 27, 2012 at 7:21 PM, Benjamin Juhn <[EMAIL PROTECTED]> wrote:
>> Hey Jon,
>>
>> I'm not seeing anything in ProtobuftoPig that is specific to handling protobuf byte fields containing serialized protobuf.
>>
>> Do you have a Pig example?
>>
>> Thanks,
>> Ben
>>
>> On Mar 26, 2012, at 4:14 PM, Jonathan Coveney wrote:
>>
>>> Look up the util class ProtobuftoPig
>>>
>>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
>>>
>>>> Thanks Jon,
>>>>
>>>> I'm using elephant-bird, and it allows me to load message A.  I'm not sure
>>>> if it has pig bindings allowing me to deserialize message B from message A
>>>> though.
>>>>
>>>> Thanks,
>>>> Ben
>>>>
>>>> On Mar 26, 2012, at 4:09 PM, Jonathan Coveney wrote:
>>>>
>>>>> Ben,
>>>>>
>>>>> take a look at elephantbird https://github.com/kevinweil/elephant-bird
>>>>>
>>>>> Jon
>>>>>
>>>>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I'm using pig with protobuf and I have some byte fields containing
>>>>>> serialized protobuf data.  Is it possible to handle this nested
>>>> serialized
>>>>>> data with pig?
>>>>>>
>>>>>> ex.
>>>>>> message A {
>>>>>> required bytes data = 1   // serialized message B
>>>>>> }
>>>>>> message B {
>>>>>> ...
>>>>>> }
>>>>>>
>>>>>> Thanks,
>>>>>> Ben
>>>>
>>>>
>>

extension are not supported yet. there is a patch pending :
https://github.com/kevinweil/elephant-bird/pull/143

Can you check if that covers your use case?

On Tue, Apr 3, 2012 at 4:32 PM, Benjamin Juhn <[EMAIL PROTECTED]> wrote:

> Thanks Dmitriy.  Doesn't look like that class supports extensions.  Am I
> missing something?
>
> - Ben
>
> On Mar 27, 2012, at 10:01 PM, Dmitriy Ryaboy wrote:
>
> > I think you want ProtobufBytesToTuple
> > (
> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/piggybank/ProtobufBytesToTuple.java
> )
> >
> > This uses ProtobufToPig to do the actual conversion, and wraps it in a
> UDF.
> >
> > D
> >
> > On Tue, Mar 27, 2012 at 7:21 PM, Benjamin Juhn <[EMAIL PROTECTED]>
> wrote:
> >> Hey Jon,
> >>
> >> I'm not seeing anything in ProtobuftoPig that is specific to handling
> protobuf byte fields containing serialized protobuf.
> >>
> >> Do you have a Pig example?
> >>
> >> Thanks,
> >> Ben
> >>
> >> On Mar 26, 2012, at 4:14 PM, Jonathan Coveney wrote:
> >>
> >>> Look up the util class ProtobuftoPig
> >>>
> >>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
> >>>
> >>>> Thanks Jon,
> >>>>
> >>>> I'm using elephant-bird, and it allows me to load message A.  I'm not
> sure
> >>>> if it has pig bindings allowing me to deserialize message B from
> message A
> >>>> though.
> >>>>
> >>>> Thanks,
> >>>> Ben
> >>>>
> >>>> On Mar 26, 2012, at 4:09 PM, Jonathan Coveney wrote:
> >>>>
> >>>>> Ben,
> >>>>>
> >>>>> take a look at elephantbird
> https://github.com/kevinweil/elephant-bird
> >>>>>
> >>>>> Jon
> >>>>>
> >>>>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> I'm using pig with protobuf and I have some byte fields containing
> >>>>>> serialized protobuf data.  Is it possible to handle this nested
> >>>> serialized
> >>>>>> data with pig?
> >>>>>>
> >>>>>> ex.
> >>>>>> message A {
> >>>>>> required bytes data = 1   // serialized message B
> >>>>>> }
> >>>>>> message B {
> >>>>>> ...
> >>>>>> }
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Ben
> >>>>
> >>>>
> >>
>
>

Looks like it's covered:
public ProtobufBytesToTuple(TypeRef<M> typeRef, ProtobufExtensionRegistry extensionRegistry) {

Thanks,
Ben

On Apr 3, 2012, at 4:41 PM, Raghu Angadi wrote:

> extension are not supported yet. there is a patch pending :
> https://github.com/kevinweil/elephant-bird/pull/143
>
> Can you check if that covers your use case?
>
> On Tue, Apr 3, 2012 at 4:32 PM, Benjamin Juhn <[EMAIL PROTECTED]> wrote:
>
>> Thanks Dmitriy.  Doesn't look like that class supports extensions.  Am I
>> missing something?
>>
>> - Ben
>>
>> On Mar 27, 2012, at 10:01 PM, Dmitriy Ryaboy wrote:
>>
>>> I think you want ProtobufBytesToTuple
>>> (
>> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/piggybank/ProtobufBytesToTuple.java
>> )
>>>
>>> This uses ProtobufToPig to do the actual conversion, and wraps it in a
>> UDF.
>>>
>>> D
>>>
>>> On Tue, Mar 27, 2012 at 7:21 PM, Benjamin Juhn <[EMAIL PROTECTED]>
>> wrote:
>>>> Hey Jon,
>>>>
>>>> I'm not seeing anything in ProtobuftoPig that is specific to handling
>> protobuf byte fields containing serialized protobuf.
>>>>
>>>> Do you have a Pig example?
>>>>
>>>> Thanks,
>>>> Ben
>>>>
>>>> On Mar 26, 2012, at 4:14 PM, Jonathan Coveney wrote:
>>>>
>>>>> Look up the util class ProtobuftoPig
>>>>>
>>>>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
>>>>>
>>>>>> Thanks Jon,
>>>>>>
>>>>>> I'm using elephant-bird, and it allows me to load message A.  I'm not
>> sure
>>>>>> if it has pig bindings allowing me to deserialize message B from
>> message A
>>>>>> though.
>>>>>>
>>>>>> Thanks,
>>>>>> Ben
>>>>>>
>>>>>> On Mar 26, 2012, at 4:09 PM, Jonathan Coveney wrote:
>>>>>>
>>>>>>> Ben,
>>>>>>>
>>>>>>> take a look at elephantbird
>> https://github.com/kevinweil/elephant-bird
>>>>>>>
>>>>>>> Jon
>>>>>>>
>>>>>>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I'm using pig with protobuf and I have some byte fields containing
>>>>>>>> serialized protobuf data.  Is it possible to handle this nested
>>>>>> serialized
>>>>>>>> data with pig?
>>>>>>>>
>>>>>>>> ex.
>>>>>>>> message A {
>>>>>>>> required bytes data = 1   // serialized message B
>>>>>>>> }
>>>>>>>> message B {
>>>>>>>> ...
>>>>>>>> }
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ben
>>>>>>
>>>>>>
>>>>
>>
>>