Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo, mail # dev - Re: Hadoop 2 compatibility issues - tangent


Copy link to this message
-
Re: Hadoop 2 compatibility issues - tangent
Christopher 2013-05-15, 02:11
With the right configuration, you could use the copy-dependencies goal
of the maven-dependency-plugin to gather your dependencies to one
place.

--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Tue, May 14, 2013 at 6:14 PM, John Vines <[EMAIL PROTECTED]> wrote:
> On that note, I was wondering if there were any suggestions for how to deal
> with the laundry list of provided dependencies that Accumulo core has?
> Writing packages against it is a bit ugly if not using the accumulo script
> to start. Are there any maven utilities to automatically dissect provided
> dependencies and make them included.
>
> Sent from my phone, please pardon the typos and brevity.
> On May 14, 2013 6:09 PM, "Keith Turner" <[EMAIL PROTECTED]> wrote:
>
>> One note about option 4.  When using 1.4 users have to include hadoop core
>> as a dependency in their pom. This must be done because the 1.4 Accumulo
>> pom marks hadoop-core as provided.  So maybe option 4 is ok if the deps in
>> the profile are provided?
>>
>>
>> On Tue, May 14, 2013 at 4:40 PM, Christopher <[EMAIL PROTECTED]> wrote:
>>
>> > So, I've run into a problem with ACCUMULO-1402 that requires a larger
>> > discussion about how Accumulo 1.5.0 should support Hadoop2.
>> >
>> > The problem is basically that profiles should not contain
>> > dependencies, because profiles don't get activated transitively. A
>> > slide deck by the Maven developers point this out as a bad practice...
>> > yet it's a practice we rely on for our current implementation of
>> > Hadoop2 support
>> > (http://www.slideshare.net/aheritier/geneva-jug-30th-march-2010-maven
>> > slide 80).
>> >
>> > What this means is that even if we go through the work of publishing
>> > binary artifacts compiled against Hadoop2, neither our Hadoop1
>> > binaries or our Hadoop2 binaries will be able to transitively resolve
>> > any dependencies defined in profiles. This has significant
>> > implications to user code that depends on Accumulo Maven artifacts.
>> > Every user will essentially have to explicitly add Hadoop dependencies
>> > for every Accumulo artifact that has dependencies on Hadoop, either
>> > because we directly or transitively depend on Hadoop (they'll have to
>> > peek into the profiles in our POMs and copy/paste the profile into
>> > their project). This becomes more complicated when we consider how
>> > users will try to use things like Instamo.
>> >
>> > There are workarounds, but none of them are really pleasant.
>> >
>> > 1. The best way to support both major Hadoop APIs is to have separate
>> > modules with separate dependencies directly in the POM. This is a fair
>> > amount of work, and in my opinion, would be too disruptive for 1.5.0.
>> > This solution also gets us separate binaries for separate supported
>> > versions, which is useful.
>> >
>> > 2. A second option, and the preferred one I think for 1.5.0, is to put
>> > a Hadoop2 patch in the branch's contrib directory
>> > (branches/1.5/contrib) that patches the POM files to support building
>> > against Hadoop2. (Acknowledgement to Keith for suggesting this
>> > solution.)
>> >
>> > 3. A third option is to fork Accumulo, and maintain two separate
>> > builds (a more traditional technique). This adds merging nightmare for
>> > features/patches, but gets around some reflection hacks that we may
>> > have been motivated to do in the past. I'm not a fan of this option,
>> > particularly because I don't want to replicate the fork nightmare that
>> > has been the history of early Hadoop itself.
>> >
>> > 4. The last option is to do nothing and to continue to build with the
>> > separate profiles as we are, and make users discover and specify
>> > transitive dependencies entirely on their own. I think this is the
>> > worst option, as it essentially amounts to "ignore the problem".
>> >
>> > At the very least, it does not seem reasonable to complete
>> > ACCUMULO-1402 for 1.5.0, given the complexity of this issue.
>> >
>> > Thoughts? Discussion? Vote on option?