Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Accumulo >> mail # dev >> Hadoop 2 compatibility issues


+
Christopher 2013-05-14, 20:40
+
Sean Busbey 2013-05-14, 20:52
+
Christopher 2013-05-14, 21:39
+
John Vines 2013-05-14, 20:56
+
Benson Margulies 2013-05-14, 21:16
Copy link to this message
-
Re: Hadoop 2 compatibility issues
I tend to agree with Sean, John, and Benson. Option 4 works for now, and
until we can define something that works better (e.g. runtime compatibility
with both hadoop 1 and 2 using reflection and crazy class loaders) we
should not delay the release. Good docs are always helpful where
engineering is less than ideal (egad, I hope I didn't just volunteer!).

Adam
On Tue, May 14, 2013 at 5:16 PM, Benson Margulies <[EMAIL PROTECTED]>wrote:

> CXF does (4) for the various competing JAX-WS implementations.
>
> The different options are API-compatible, and the profiles just switch
> the deps around.
>
> There would be slightly more Maven correctness in marking the deps
> optional, forcing each user to pick one explicitly.
>
> However, (4) with good doc on what to put in the POM is really not a
> cause for shame. Maven is weak in this area, and it's all tradeoffs.
>
>
>
> On Tue, May 14, 2013 at 4:56 PM, John Vines <[EMAIL PROTECTED]> wrote:
> > I'm an advocate of option 4. You say that it's ignoring the problem,
> > whereas I think it's waiting until we have the time to solve the problem
> > correctly. Your reasoning for this is for standardizing for maven
> > conventions, but the other options, while more 'correct' from a maven
> > standpoint or a larger headache for our user base and ourselves. In
> either
> > case, we're going to be breaking some sort of convention, and while it's
> > not good, we should be doing the one that's less bad for US. The
> important
> > thing here, now, is that the poms work and we should go with the method
> > that leaves the work minimal for our end users to utilize them.
> >
> > I do agree that 1. is the correct option in the long run. More
> > specifically, I think it boils down to having a single module
> compatibility
> > layer, which is how hbase deals with this issue. But like you said, we
> > don't have the time to engineer a proper solution. So let sleeping dogs
> lie
> > and we can revamp the whole system for 1.5.1 or 1.6.0 when we have the
> > cycles to do it right.
> >
> >
> > On Tue, May 14, 2013 at 4:40 PM, Christopher <[EMAIL PROTECTED]>
> wrote:
> >
> >> So, I've run into a problem with ACCUMULO-1402 that requires a larger
> >> discussion about how Accumulo 1.5.0 should support Hadoop2.
> >>
> >> The problem is basically that profiles should not contain
> >> dependencies, because profiles don't get activated transitively. A
> >> slide deck by the Maven developers point this out as a bad practice...
> >> yet it's a practice we rely on for our current implementation of
> >> Hadoop2 support
> >> (http://www.slideshare.net/aheritier/geneva-jug-30th-march-2010-maven
> >> slide 80).
> >>
> >> What this means is that even if we go through the work of publishing
> >> binary artifacts compiled against Hadoop2, neither our Hadoop1
> >> binaries or our Hadoop2 binaries will be able to transitively resolve
> >> any dependencies defined in profiles. This has significant
> >> implications to user code that depends on Accumulo Maven artifacts.
> >> Every user will essentially have to explicitly add Hadoop dependencies
> >> for every Accumulo artifact that has dependencies on Hadoop, either
> >> because we directly or transitively depend on Hadoop (they'll have to
> >> peek into the profiles in our POMs and copy/paste the profile into
> >> their project). This becomes more complicated when we consider how
> >> users will try to use things like Instamo.
> >>
> >> There are workarounds, but none of them are really pleasant.
> >>
> >> 1. The best way to support both major Hadoop APIs is to have separate
> >> modules with separate dependencies directly in the POM. This is a fair
> >> amount of work, and in my opinion, would be too disruptive for 1.5.0.
> >> This solution also gets us separate binaries for separate supported
> >> versions, which is useful.
> >>
> >> 2. A second option, and the preferred one I think for 1.5.0, is to put
> >> a Hadoop2 patch in the branch's contrib directory
> >> (branches/1.5/contrib) that patches the POM files to support building
+
Christopher 2013-05-14, 21:42
+
Christopher 2013-05-14, 21:48
+
Benson Margulies 2013-05-14, 21:51
+
Keith Turner 2013-05-14, 22:13
+
John Vines 2013-05-14, 22:16
+
Benson Margulies 2013-05-14, 23:09
+
John Vines 2013-05-14, 23:43
+
Christopher 2013-05-14, 23:36
+
Benson Margulies 2013-05-14, 23:41
+
Christopher 2013-05-14, 23:47
+
John Vines 2013-05-14, 23:50
+
Benson Margulies 2013-05-15, 00:27
+
Christopher 2013-05-15, 01:01
+
Adam Fuchs 2013-05-15, 21:31
+
Christopher 2013-05-15, 21:44
+
John Vines 2013-05-16, 00:03
+
Eric Newton 2013-05-16, 15:23
+
Adam Fuchs 2013-05-16, 15:51
+
John Vines 2013-05-14, 23:46
+
Josh Elser 2013-05-14, 23:36
+
John Vines 2013-05-14, 21:59
+
John Vines 2013-05-14, 21:52
+
Keith Turner 2013-05-14, 22:08
+
Sean Busbey 2013-05-14, 22:14
+
Eric Newton 2013-05-14, 23:23
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB