Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> protobuf 2.4.1 and ObjectInspector


Copy link to this message
-
Re: protobuf 2.4.1 and ObjectInspector
Thanks Ed for your reply on this.

I specifically have been working on enhancing the existing hbase hive
integration to handle advanced proto/thrift structures and also logged [1]
in the process. I wasn't able to use the existing
ProtocolBuffersObjectInspector for this purpose as they extend the
ReflectionStructObjectInspector which I have seen to fail in some cases. I
decided to provided the struct schema explicitly in the CREATE EXTERNAL
TABLE command itself and then use that to lazily deserialize the structure.

This approach worked well for most cases until I hit more complicated cases
like circular references and nested repeated messages in protobufs. I
suspect that reflections "might" have worked better for such complicated
cases.

You mentioned writing a serde that takes the protobuf objects directly and
deserialize them. Did you mean to use reflections or follow a method
similar to one I described above? Any suggestions?

Thanks,

On Tue, May 22, 2012 at 9:00 PM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> I am trying to decipher my way though the code as well. Apparently the
> work is half done to return ProfotBuf as a native type. As there is
> support for ObjectInstector.THRIFT and ObjectInspector.ProtoBuffer. I
> currently want to write a Serde that works like the thrift serde where
> protobuf objects can be given directly to hive. Come hang out in the
> IRC room and maybe we can chat more about this.
>
> On Tue, May 22, 2012 at 6:09 PM, [EMAIL PROTECTED]
> <[EMAIL PROTECTED]> wrote:
> > I am trying to use the ReflectionStructObjectInspector to extract fields
> > from a protobuf generated from 2.4.1 compiler. I am seeing that
> reflections
> > fails to extract fields out of the generated protobuf class.
> Specifically,
> > this code snippet:
> >
> > public static Field[] getDeclaredNonStaticFields(Class<?> c) {
> >
> >     Field[] f = c.getDeclaredFields();    // This returns back the
> correct
> > number of fields
> >
> >     ArrayList<Field> af = new ArrayList<Field>();
> >
> >     for (int i = 0; i < f.length; ++i) {
> >
> >       // The logic here falls flat as it is looking only for the
> non-static
> > fields and all generated fields
> >
> >      // seem to be static
> >
> >       if (!Modifier.isStatic(f[i].getModifiers())) {
> >
> >         af.add(f[i]);
> >
> >       }
> >
> >     }
> >
> >     Field[] r = new Field[af.size()];
> >
> >     for (int i = 0; i < af.size(); ++i) {
> >
> >       r[i] = af.get(i);
> >
> >     }
> >
> >     return r;
> >
> >   }
> >
> >
> > This causes the whole ObjectInspector to fail. Has anyone else seen this
> > issue too?
> >
>

--
Swarnim
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB