Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo >> mail # dev >> Re: Hadoop 2 compatibility issues - tangent


+
John Vines 2013-05-14, 22:14
+
Christopher 2013-05-15, 02:11
Copy link to this message
-
Re: Hadoop 2 compatibility issues - tangent
Awesome Chris, thanks. I didn't know where to begin looking for that one.

Sent from my phone, please pardon the typos and brevity.
On May 14, 2013 7:11 PM, "Christopher" <[EMAIL PROTECTED]> wrote:

> With the right configuration, you could use the copy-dependencies goal
> of the maven-dependency-plugin to gather your dependencies to one
> place.
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Tue, May 14, 2013 at 6:14 PM, John Vines <[EMAIL PROTECTED]> wrote:
> > On that note, I was wondering if there were any suggestions for how to
> deal
> > with the laundry list of provided dependencies that Accumulo core has?
> > Writing packages against it is a bit ugly if not using the accumulo
> script
> > to start. Are there any maven utilities to automatically dissect provided
> > dependencies and make them included.
> >
> > Sent from my phone, please pardon the typos and brevity.
> > On May 14, 2013 6:09 PM, "Keith Turner" <[EMAIL PROTECTED]> wrote:
> >
> >> One note about option 4.  When using 1.4 users have to include hadoop
> core
> >> as a dependency in their pom. This must be done because the 1.4 Accumulo
> >> pom marks hadoop-core as provided.  So maybe option 4 is ok if the deps
> in
> >> the profile are provided?
> >>
> >>
> >> On Tue, May 14, 2013 at 4:40 PM, Christopher <[EMAIL PROTECTED]>
> wrote:
> >>
> >> > So, I've run into a problem with ACCUMULO-1402 that requires a larger
> >> > discussion about how Accumulo 1.5.0 should support Hadoop2.
> >> >
> >> > The problem is basically that profiles should not contain
> >> > dependencies, because profiles don't get activated transitively. A
> >> > slide deck by the Maven developers point this out as a bad practice...
> >> > yet it's a practice we rely on for our current implementation of
> >> > Hadoop2 support
> >> > (http://www.slideshare.net/aheritier/geneva-jug-30th-march-2010-maven
> >> > slide 80).
> >> >
> >> > What this means is that even if we go through the work of publishing
> >> > binary artifacts compiled against Hadoop2, neither our Hadoop1
> >> > binaries or our Hadoop2 binaries will be able to transitively resolve
> >> > any dependencies defined in profiles. This has significant
> >> > implications to user code that depends on Accumulo Maven artifacts.
> >> > Every user will essentially have to explicitly add Hadoop dependencies
> >> > for every Accumulo artifact that has dependencies on Hadoop, either
> >> > because we directly or transitively depend on Hadoop (they'll have to
> >> > peek into the profiles in our POMs and copy/paste the profile into
> >> > their project). This becomes more complicated when we consider how
> >> > users will try to use things like Instamo.
> >> >
> >> > There are workarounds, but none of them are really pleasant.
> >> >
> >> > 1. The best way to support both major Hadoop APIs is to have separate
> >> > modules with separate dependencies directly in the POM. This is a fair
> >> > amount of work, and in my opinion, would be too disruptive for 1.5.0.
> >> > This solution also gets us separate binaries for separate supported
> >> > versions, which is useful.
> >> >
> >> > 2. A second option, and the preferred one I think for 1.5.0, is to put
> >> > a Hadoop2 patch in the branch's contrib directory
> >> > (branches/1.5/contrib) that patches the POM files to support building
> >> > against Hadoop2. (Acknowledgement to Keith for suggesting this
> >> > solution.)
> >> >
> >> > 3. A third option is to fork Accumulo, and maintain two separate
> >> > builds (a more traditional technique). This adds merging nightmare for
> >> > features/patches, but gets around some reflection hacks that we may
> >> > have been motivated to do in the past. I'm not a fan of this option,
> >> > particularly because I don't want to replicate the fork nightmare that
> >> > has been the history of early Hadoop itself.
> >> >
> >> > 4. The last option is to do nothing and to continue to build with the
> >> > separate profiles as we are, and make users discover and specify
+
Christopher 2013-05-15, 16:33
+
David Medinets 2013-05-15, 01:40