Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> deserializing nested protobufs


+
Benjamin Juhn 2012-03-26, 22:30
+
Jonathan Coveney 2012-03-26, 23:09
+
Benjamin Juhn 2012-03-26, 23:11
+
Jonathan Coveney 2012-03-26, 23:14
+
Benjamin Juhn 2012-03-28, 02:21
+
Dmitriy Ryaboy 2012-03-28, 05:01
+
Benjamin Juhn 2012-04-03, 23:32
+
Raghu Angadi 2012-04-03, 23:41
Copy link to this message
-
Re: deserializing nested protobufs
Looks like it's covered:
public ProtobufBytesToTuple(TypeRef<M> typeRef, ProtobufExtensionRegistry extensionRegistry) {

Thanks,
Ben

On Apr 3, 2012, at 4:41 PM, Raghu Angadi wrote:

> extension are not supported yet. there is a patch pending :
> https://github.com/kevinweil/elephant-bird/pull/143
>
> Can you check if that covers your use case?
>
> On Tue, Apr 3, 2012 at 4:32 PM, Benjamin Juhn <[EMAIL PROTECTED]> wrote:
>
>> Thanks Dmitriy.  Doesn't look like that class supports extensions.  Am I
>> missing something?
>>
>> - Ben
>>
>> On Mar 27, 2012, at 10:01 PM, Dmitriy Ryaboy wrote:
>>
>>> I think you want ProtobufBytesToTuple
>>> (
>> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/piggybank/ProtobufBytesToTuple.java
>> )
>>>
>>> This uses ProtobufToPig to do the actual conversion, and wraps it in a
>> UDF.
>>>
>>> D
>>>
>>> On Tue, Mar 27, 2012 at 7:21 PM, Benjamin Juhn <[EMAIL PROTECTED]>
>> wrote:
>>>> Hey Jon,
>>>>
>>>> I'm not seeing anything in ProtobuftoPig that is specific to handling
>> protobuf byte fields containing serialized protobuf.
>>>>
>>>> Do you have a Pig example?
>>>>
>>>> Thanks,
>>>> Ben
>>>>
>>>> On Mar 26, 2012, at 4:14 PM, Jonathan Coveney wrote:
>>>>
>>>>> Look up the util class ProtobuftoPig
>>>>>
>>>>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
>>>>>
>>>>>> Thanks Jon,
>>>>>>
>>>>>> I'm using elephant-bird, and it allows me to load message A.  I'm not
>> sure
>>>>>> if it has pig bindings allowing me to deserialize message B from
>> message A
>>>>>> though.
>>>>>>
>>>>>> Thanks,
>>>>>> Ben
>>>>>>
>>>>>> On Mar 26, 2012, at 4:09 PM, Jonathan Coveney wrote:
>>>>>>
>>>>>>> Ben,
>>>>>>>
>>>>>>> take a look at elephantbird
>> https://github.com/kevinweil/elephant-bird
>>>>>>>
>>>>>>> Jon
>>>>>>>
>>>>>>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I'm using pig with protobuf and I have some byte fields containing
>>>>>>>> serialized protobuf data.  Is it possible to handle this nested
>>>>>> serialized
>>>>>>>> data with pig?
>>>>>>>>
>>>>>>>> ex.
>>>>>>>> message A {
>>>>>>>> required bytes data = 1   // serialized message B
>>>>>>>> }
>>>>>>>> message B {
>>>>>>>> ...
>>>>>>>> }
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ben
>>>>>>
>>>>>>
>>>>
>>
>>

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB