Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> deserializing nested protobufs


Copy link to this message
-
Re: deserializing nested protobufs
Looks like it's covered:
public ProtobufBytesToTuple(TypeRef<M> typeRef, ProtobufExtensionRegistry extensionRegistry) {

Thanks,
Ben

On Apr 3, 2012, at 4:41 PM, Raghu Angadi wrote:

> extension are not supported yet. there is a patch pending :
> https://github.com/kevinweil/elephant-bird/pull/143
>
> Can you check if that covers your use case?
>
> On Tue, Apr 3, 2012 at 4:32 PM, Benjamin Juhn <[EMAIL PROTECTED]> wrote:
>
>> Thanks Dmitriy.  Doesn't look like that class supports extensions.  Am I
>> missing something?
>>
>> - Ben
>>
>> On Mar 27, 2012, at 10:01 PM, Dmitriy Ryaboy wrote:
>>
>>> I think you want ProtobufBytesToTuple
>>> (
>> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/piggybank/ProtobufBytesToTuple.java
>> )
>>>
>>> This uses ProtobufToPig to do the actual conversion, and wraps it in a
>> UDF.
>>>
>>> D
>>>
>>> On Tue, Mar 27, 2012 at 7:21 PM, Benjamin Juhn <[EMAIL PROTECTED]>
>> wrote:
>>>> Hey Jon,
>>>>
>>>> I'm not seeing anything in ProtobuftoPig that is specific to handling
>> protobuf byte fields containing serialized protobuf.
>>>>
>>>> Do you have a Pig example?
>>>>
>>>> Thanks,
>>>> Ben
>>>>
>>>> On Mar 26, 2012, at 4:14 PM, Jonathan Coveney wrote:
>>>>
>>>>> Look up the util class ProtobuftoPig
>>>>>
>>>>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
>>>>>
>>>>>> Thanks Jon,
>>>>>>
>>>>>> I'm using elephant-bird, and it allows me to load message A.  I'm not
>> sure
>>>>>> if it has pig bindings allowing me to deserialize message B from
>> message A
>>>>>> though.
>>>>>>
>>>>>> Thanks,
>>>>>> Ben
>>>>>>
>>>>>> On Mar 26, 2012, at 4:09 PM, Jonathan Coveney wrote:
>>>>>>
>>>>>>> Ben,
>>>>>>>
>>>>>>> take a look at elephantbird
>> https://github.com/kevinweil/elephant-bird
>>>>>>>
>>>>>>> Jon
>>>>>>>
>>>>>>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I'm using pig with protobuf and I have some byte fields containing
>>>>>>>> serialized protobuf data.  Is it possible to handle this nested
>>>>>> serialized
>>>>>>>> data with pig?
>>>>>>>>
>>>>>>>> ex.
>>>>>>>> message A {
>>>>>>>> required bytes data = 1   // serialized message B
>>>>>>>> }
>>>>>>>> message B {
>>>>>>>> ...
>>>>>>>> }
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ben
>>>>>>
>>>>>>
>>>>
>>
>>