-
Re: Publishing jars for hbase compiled against hadoop 0.23.x/hadoop 2.0.xJesse Yates 2012-05-16, 22:22
Comments inline.
TL;DR
+1 on a small number of supported versions with different classifiers that
only span a limited api skew to avoid a mountain of reflection. Along with
that, support for the builds via jenkins testing.
Any further dependency resoluton should be considered 'external projects'
and handled via their own maven setttings.xml which can be in external
repos by people who want hbase to support other versions of our
dependencies (and possibly have a branch of hbase with the appropriate
modifications). Any new dependency versions we want to support should be
heavily vetted for ease of integration and stability.
-1 on keeping code in the POMs for things we don't directly release as that
means more potential maintaince for things we (as a community) don't care
that much about (ala current Avro support).
-------------------
Jesse Yates
@jesse_yates
jyates.github.com
On Wed, May 16, 2012 at 2:09 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
> On Wed, May 16, 2012 at 1:00 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
> > Now, it seems like in Bigtop we're going to soon expose the Maven repo
> > with all of the Maven artifacts constituting a particular Bigtop
> "stack". You
> > could think of it as a transitive closure of all of the deps. built
> against
> > each other. This, of course, will not tackle an issue of a random
> combination
> > of components (we only support the versions of components as
> > specified in our own BOM for each particular Bigtop release) but it will
> > provide a pretty stable body of Maven artifacts that are KNOWN (as
> > in tested) to be compiled against each other.
>
> I think HBase should consider having a single blessed set of
> dependencies and only one build for a given release,
This would be really nice, but seems a bit unreasonable given that we are
the "hadoop database" (if not in name, at least by connotation). I think
limiting our support to the latest X versions (2-3?) is reasonable given
consistent APIs - we should be very careful in picking which new versions
we support and when. A lot of the pain with the hadoop distributions has
been then wildly shifting apis, making a lot of work painful for handling
different versions (distcp/backup situations come to mind here, among other
things.
> but also several
> Jenkins projects set up to insure that release also builds against
> some larger set of additional dependencies according to contributor
> needs,
Definitely a necessity if we support more than 1 version. Only problem here
is that we then have to worry about multiple builds, which seemed to be a
problem in the past. If we are going to support more than 1 version, we
need to have full support for that version/permutation of options (eg.
Hadoop X with Zookeeper Y)
> and otherwise the user is welcome to mvn
> -Ddependency.version=foo.
I'd prefer not to have pieces in the code that are not being regularly
tested/used. If we find we have a lot of people using a given version and
willing to support it, then we should roll it in (like with other external
dependencies, like the Avro stuff that we are stuck with).
The mvn command you recommend above is already quite close to what we are
doing already, with just specifying the hadoop version as a profile, eg (or
close enough)
-Dhadoop.version=0.23
+1 on the idea of having classifiers for the different versions we actually
release as proper artifacts, and should be completely reasonable to enable
via profiles. I'd have to double check as to _how_ people would specify
that classifier/version of hbase from the maven repo, but it seems entirely
possible (my worry here is about the collison with the -tests and -sources
classifiers, which are standard mvn conventions for different builds).
Otherwise, with maven it is very reasonable to have people hosting profiles
for versions that they want to support - generally, this means just another
settings.xml file that includes another profile that people can activate on
their own, when they want to build against their own version.
TL;DR
+1 on a small number of supported versions with different classifiers that
only span a limited api skew to avoid a mountain of reflection. Along with
that, support for the builds via jenkins testing.
Any further dependency resoluton should be considered 'external projects'
and handled via their own maven setttings.xml which can be in external
repos by people who want hbase to support other versions of our
dependencies (and possibly have a branch of hbase with the appropriate
modifications). Any new dependency versions we want to support should be
heavily vetted for ease of integration and stability.
-1 on keeping code in the POMs for things we don't directly release as that
means more potential maintaince for things we (as a community) don't care
that much about (ala current Avro support).
-------------------
Jesse Yates
@jesse_yates
jyates.github.com
On Wed, May 16, 2012 at 2:09 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
> On Wed, May 16, 2012 at 1:00 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
> > Now, it seems like in Bigtop we're going to soon expose the Maven repo
> > with all of the Maven artifacts constituting a particular Bigtop
> "stack". You
> > could think of it as a transitive closure of all of the deps. built
> against
> > each other. This, of course, will not tackle an issue of a random
> combination
> > of components (we only support the versions of components as
> > specified in our own BOM for each particular Bigtop release) but it will
> > provide a pretty stable body of Maven artifacts that are KNOWN (as
> > in tested) to be compiled against each other.
>
> I think HBase should consider having a single blessed set of
> dependencies and only one build for a given release,
This would be really nice, but seems a bit unreasonable given that we are
the "hadoop database" (if not in name, at least by connotation). I think
limiting our support to the latest X versions (2-3?) is reasonable given
consistent APIs - we should be very careful in picking which new versions
we support and when. A lot of the pain with the hadoop distributions has
been then wildly shifting apis, making a lot of work painful for handling
different versions (distcp/backup situations come to mind here, among other
things.
> but also several
> Jenkins projects set up to insure that release also builds against
> some larger set of additional dependencies according to contributor
> needs,
Definitely a necessity if we support more than 1 version. Only problem here
is that we then have to worry about multiple builds, which seemed to be a
problem in the past. If we are going to support more than 1 version, we
need to have full support for that version/permutation of options (eg.
Hadoop X with Zookeeper Y)
> and otherwise the user is welcome to mvn
> -Ddependency.version=foo.
I'd prefer not to have pieces in the code that are not being regularly
tested/used. If we find we have a lot of people using a given version and
willing to support it, then we should roll it in (like with other external
dependencies, like the Avro stuff that we are stuck with).
The mvn command you recommend above is already quite close to what we are
doing already, with just specifying the hadoop version as a profile, eg (or
close enough)
-Dhadoop.version=0.23
+1 on the idea of having classifiers for the different versions we actually
release as proper artifacts, and should be completely reasonable to enable
via profiles. I'd have to double check as to _how_ people would specify
that classifier/version of hbase from the maven repo, but it seems entirely
possible (my worry here is about the collison with the -tests and -sources
classifiers, which are standard mvn conventions for different builds).
Otherwise, with maven it is very reasonable to have people hosting profiles
for versions that they want to support - generally, this means just another
settings.xml file that includes another profile that people can activate on
their own, when they want to build against their own version.
