Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill >> mail # dev >> problems with protoc versions / anyone tried under hadoop 2.0?


Copy link to this message
-
Re: problems with protoc versions / anyone tried under hadoop 2.0?
I think I might have put together a way to make things work as you said,
with maven-shade-plugin. Will verify tomorrow.

The idea is to rename classes in protobuf-java and references to pb-java in
'common' into renamed package '250.com.google.protobuf.*' and to package
together into an alternative jar: common-1.0-SNAPSHOT-rebuffed.jar

The compilation of java-exec would depend on normal common-1.0-SNAPSHOT.jar
and normal protobuf-java without a problem. java-exec also packages an
alternative jar: java-exec-1.0-SNAPSHOT-rebuffed.jar, with all references
to pb-java renamed, but without renamed pb-java itself.

In this way nothing changed under normal workflow; If one wants to run with
hadoop 2.0, he/she can have dependency on rebuffed jars with
<classifier>rebuffed</classifier> .

the patch is at: https://issues.apache.org/jira/browse/DRILL-161

Is there any problem? Or any better way to do this?
On Fri, Aug 2, 2013 at 3:00 AM, Jacques Nadeau <[EMAIL PROTECTED]> wrote:

> Yeah...
>
> It's painful that 2.5 and 2.4.1 don't coeexist better together.  It
> seems like they should simply separate the classes that are bw/fw
> compatible and those that aren't.  C'est la vie.
>
> My current thinking is that we generate an alternate 2.5 jar that
> modifies all the package names and then use a mvn filter on the
> generated protoc code as part of the compile part of the build.  I
> much prefer doing it compile time rather than trying to fight through
> classloader issues.  The last piece is modifying the protoc builds to
> be allowed to be pointed at an alternative path for the protoc
> generator so people can have both on their system (if people want to
> build a combination of Drill and Hadoop or HBase).
>
> But I haven't gotten to it yet.  We should get a jira in for it and
> see if a maven maestro can help us out.
>
> J
>
> On Wed, Jul 31, 2013 at 12:17 AM, Lisen Mu <[EMAIL PROTECTED]> wrote:
> > Hi drillers,
> >
> > Has anyone tried scan under hadoop 2.0?
> >
> > We've built a HBase scanner, with hadoop 2.0.x.
> >
> > The problem is, drill depends on protoc 2.5.0, while hadoop 2.0.x
> currently
> > depends on 2.4.x.
> >
> > Is there any suggestion to keep both drill and hadoop 2.0 working
> together?
> >
> > Keeping a branch of drill with protoc downgraded to 2.4 is just painful.
> It
> > would be great if the drill master could downgrade to 2.4, however, the
> > most promising solution is likely to be some class loading techniques to
> > keep both happy. Additional requirement is to keep 'mvn test' happy, too.
> > Any suggestion on this? What's the easiest way?
> >
> > Thanks,
> >
> > Lisen
>