Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill >> mail # dev >> problems with protoc versions / anyone tried under hadoop 2.0?


Copy link to this message
-
Re: problems with protoc versions / anyone tried under hadoop 2.0?
I think I might have put together a way to make things work as you said,
with maven-shade-plugin. Will verify tomorrow.

The idea is to rename classes in protobuf-java and references to pb-java in
'common' into renamed package '250.com.google.protobuf.*' and to package
together into an alternative jar: common-1.0-SNAPSHOT-rebuffed.jar

The compilation of java-exec would depend on normal common-1.0-SNAPSHOT.jar
and normal protobuf-java without a problem. java-exec also packages an
alternative jar: java-exec-1.0-SNAPSHOT-rebuffed.jar, with all references
to pb-java renamed, but without renamed pb-java itself.

In this way nothing changed under normal workflow; If one wants to run with
hadoop 2.0, he/she can have dependency on rebuffed jars with
<classifier>rebuffed</classifier> .

the patch is at: https://issues.apache.org/jira/browse/DRILL-161

Is there any problem? Or any better way to do this?
On Fri, Aug 2, 2013 at 3:00 AM, Jacques Nadeau <[EMAIL PROTECTED]> wrote:

> Yeah...
>
> It's painful that 2.5 and 2.4.1 don't coeexist better together.  It
> seems like they should simply separate the classes that are bw/fw
> compatible and those that aren't.  C'est la vie.
>
> My current thinking is that we generate an alternate 2.5 jar that
> modifies all the package names and then use a mvn filter on the
> generated protoc code as part of the compile part of the build.  I
> much prefer doing it compile time rather than trying to fight through
> classloader issues.  The last piece is modifying the protoc builds to
> be allowed to be pointed at an alternative path for the protoc
> generator so people can have both on their system (if people want to
> build a combination of Drill and Hadoop or HBase).
>
> But I haven't gotten to it yet.  We should get a jira in for it and
> see if a maven maestro can help us out.
>
> J
>
> On Wed, Jul 31, 2013 at 12:17 AM, Lisen Mu <[EMAIL PROTECTED]> wrote:
> > Hi drillers,
> >
> > Has anyone tried scan under hadoop 2.0?
> >
> > We've built a HBase scanner, with hadoop 2.0.x.
> >
> > The problem is, drill depends on protoc 2.5.0, while hadoop 2.0.x
> currently
> > depends on 2.4.x.
> >
> > Is there any suggestion to keep both drill and hadoop 2.0 working
> together?
> >
> > Keeping a branch of drill with protoc downgraded to 2.4 is just painful.
> It
> > would be great if the drill master could downgrade to 2.4, however, the
> > most promising solution is likely to be some class loading techniques to
> > keep both happy. Additional requirement is to keep 'mvn test' happy, too.
> > Any suggestion on this? What's the easiest way?
> >
> > Thanks,
> >
> > Lisen
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB