Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> deserializing nested protobufs


Copy link to this message
-
Re: deserializing nested protobufs
extension are not supported yet. there is a patch pending :
https://github.com/kevinweil/elephant-bird/pull/143

Can you check if that covers your use case?

On Tue, Apr 3, 2012 at 4:32 PM, Benjamin Juhn <[EMAIL PROTECTED]> wrote:

> Thanks Dmitriy.  Doesn't look like that class supports extensions.  Am I
> missing something?
>
> - Ben
>
> On Mar 27, 2012, at 10:01 PM, Dmitriy Ryaboy wrote:
>
> > I think you want ProtobufBytesToTuple
> > (
> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/piggybank/ProtobufBytesToTuple.java
> )
> >
> > This uses ProtobufToPig to do the actual conversion, and wraps it in a
> UDF.
> >
> > D
> >
> > On Tue, Mar 27, 2012 at 7:21 PM, Benjamin Juhn <[EMAIL PROTECTED]>
> wrote:
> >> Hey Jon,
> >>
> >> I'm not seeing anything in ProtobuftoPig that is specific to handling
> protobuf byte fields containing serialized protobuf.
> >>
> >> Do you have a Pig example?
> >>
> >> Thanks,
> >> Ben
> >>
> >> On Mar 26, 2012, at 4:14 PM, Jonathan Coveney wrote:
> >>
> >>> Look up the util class ProtobuftoPig
> >>>
> >>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
> >>>
> >>>> Thanks Jon,
> >>>>
> >>>> I'm using elephant-bird, and it allows me to load message A.  I'm not
> sure
> >>>> if it has pig bindings allowing me to deserialize message B from
> message A
> >>>> though.
> >>>>
> >>>> Thanks,
> >>>> Ben
> >>>>
> >>>> On Mar 26, 2012, at 4:09 PM, Jonathan Coveney wrote:
> >>>>
> >>>>> Ben,
> >>>>>
> >>>>> take a look at elephantbird
> https://github.com/kevinweil/elephant-bird
> >>>>>
> >>>>> Jon
> >>>>>
> >>>>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> I'm using pig with protobuf and I have some byte fields containing
> >>>>>> serialized protobuf data.  Is it possible to handle this nested
> >>>> serialized
> >>>>>> data with pig?
> >>>>>>
> >>>>>> ex.
> >>>>>> message A {
> >>>>>> required bytes data = 1   // serialized message B
> >>>>>> }
> >>>>>> message B {
> >>>>>> ...
> >>>>>> }
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Ben
> >>>>
> >>>>
> >>
>
>