Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> deserializing nested protobufs


Copy link to this message
-
Re: deserializing nested protobufs
extension are not supported yet. there is a patch pending :
https://github.com/kevinweil/elephant-bird/pull/143

Can you check if that covers your use case?

On Tue, Apr 3, 2012 at 4:32 PM, Benjamin Juhn <[EMAIL PROTECTED]> wrote:

> Thanks Dmitriy.  Doesn't look like that class supports extensions.  Am I
> missing something?
>
> - Ben
>
> On Mar 27, 2012, at 10:01 PM, Dmitriy Ryaboy wrote:
>
> > I think you want ProtobufBytesToTuple
> > (
> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/piggybank/ProtobufBytesToTuple.java
> )
> >
> > This uses ProtobufToPig to do the actual conversion, and wraps it in a
> UDF.
> >
> > D
> >
> > On Tue, Mar 27, 2012 at 7:21 PM, Benjamin Juhn <[EMAIL PROTECTED]>
> wrote:
> >> Hey Jon,
> >>
> >> I'm not seeing anything in ProtobuftoPig that is specific to handling
> protobuf byte fields containing serialized protobuf.
> >>
> >> Do you have a Pig example?
> >>
> >> Thanks,
> >> Ben
> >>
> >> On Mar 26, 2012, at 4:14 PM, Jonathan Coveney wrote:
> >>
> >>> Look up the util class ProtobuftoPig
> >>>
> >>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
> >>>
> >>>> Thanks Jon,
> >>>>
> >>>> I'm using elephant-bird, and it allows me to load message A.  I'm not
> sure
> >>>> if it has pig bindings allowing me to deserialize message B from
> message A
> >>>> though.
> >>>>
> >>>> Thanks,
> >>>> Ben
> >>>>
> >>>> On Mar 26, 2012, at 4:09 PM, Jonathan Coveney wrote:
> >>>>
> >>>>> Ben,
> >>>>>
> >>>>> take a look at elephantbird
> https://github.com/kevinweil/elephant-bird
> >>>>>
> >>>>> Jon
> >>>>>
> >>>>> 2012/3/26 Benjamin Juhn <[EMAIL PROTECTED]>
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> I'm using pig with protobuf and I have some byte fields containing
> >>>>>> serialized protobuf data.  Is it possible to handle this nested
> >>>> serialized
> >>>>>> data with pig?
> >>>>>>
> >>>>>> ex.
> >>>>>> message A {
> >>>>>> required bytes data = 1   // serialized message B
> >>>>>> }
> >>>>>> message B {
> >>>>>> ...
> >>>>>> }
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Ben
> >>>>
> >>>>
> >>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB