Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Publishing jars for hbase compiled against hadoop 0.23.x/hadoop 2.0.x


+
Jonathan Hsieh 2012-05-16, 18:24
+
Andrew Purtell 2012-05-16, 18:27
+
Jonathan Hsieh 2012-05-16, 18:35
+
Gary Helmling 2012-05-16, 19:45
+
Alejandro Abdelnur 2012-05-16, 19:50
+
Roman Shaposhnik 2012-05-16, 20:00
+
Andrew Purtell 2012-05-16, 21:09
+
Jesse Yates 2012-05-16, 22:22
+
Andrew Purtell 2012-05-16, 23:06
+
Alejandro Abdelnur 2012-05-16, 23:17
Copy link to this message
-
Re: Publishing jars for hbase compiled against hadoop 0.23.x/hadoop 2.0.x
See my comments inlined...

On Wed, May 16, 2012 at 04:06PM, Andrew Purtell wrote:
> [cc bigtop-dev]
>
> On Wed, May 16, 2012 at 3:22 PM, Jesse Yates <[EMAIL PROTECTED]> wrote:
> > ═+1 on a small number of supported versions with different classifiers that
> > only span a limited api skew to avoid a mountain of reflection. Along with
> > that, support for the builds via jenkins testing.
>
> and
>
> >> I think HBase should consider having a single blessed set of
> >> dependencies and only one build for a given release,
> >
> > This would be really nice, but seems a bit unreasonable given that we are
> > the "hadoop database" (if not in name, at least by connotation). I think
> > limiting our support to the latest X versions (2-3?) is reasonable given
> > consistent APIs
>
> I was talking release mechanics not source/compilation/testing level
> support. Hence the suggestion for multiple Jenkins projects for the
> dependency versions we care about. That care could be scoped like you
> suggest.
>
> I like what Bigtop espouses: carefully constructed snapshots of the
> world, well tested in total. Seems easier to manage then laying out
> various planes from increasingly higher dimensional spaces. If they
> get traction we can act as a responsible upstream project. As for our
> official release, we'd have maybe two, I'll grant you that, Hadoop 1
> and Hadoop 2.
>
> X=2 will be a challenge. It's not just the Hadoop version that could
> change, but the versions of all of its dependencies, SLF4J, Guava,
> JUnit, protobuf, etc. etc. etc.; and that could happen at any time on
> point releases. If we are supporting the whole series of 1.x and 2.x
> releases, then that could be a real pain. Guava is a good example, it
> was a bit painful for us to move from 9 to 11 but not so for core as
> far as I know.

One of the by-design advantages of stack-assembly-validation automation
approach (that BigTop incidentally took ;) is that it provides a relatively
no-effort creation of stack updates when a single or multiple dependencies got
changed. Yes, it requires certain upfront time-investment to make the first
base stack definition. And from here it should be pretty much downhill.

We have applied a similar approach for the creation of X86 Solaris based
stacks for Sun Microsystems' rack-mount servers and it was a hoot and saved us
a tremendous amount of money back then (not that it helped Sun in the long
run)

>  - we should be very careful in picking which new versions
> > we support and when. A lot of the pain with the hadoop distributions has
> > been then wildly shifting apis, making a lot of work painful for handling
> > different versions (distcp/backup situations come to mind here, among other
> > things.
>
> We also have test dependencies on interfaces that are LimitedPrivate
> at best. It's a source of friction.
>
> > +1 on the idea of having classifiers for the different versions we actually
> > release as proper artifacts, and should be completely reasonable to enable
> > via profiles. I'd have to double check as to _how_ people would specify
> > that classifier/version of hbase from the maven repo, but it seems entirely
> > possible (my worry here is about the collison with the -tests and -sources
> > classifiers, which are standard mvn conventions for different builds).
> > Otherwise, with maven it is very reasonable to have people hosting profiles
> > for versions that they want to support - generally, this means just another
> > settings.xml file that includes another profile that people can activate on
> > their own, when they want to build against their own version.
>
> This was a question I had, maybe you know. What happens if you want to
> build something like <artifact>-<version>-<classifier>-tests or
> -source? Would that work? Otherwise we'd have to add a suffix using
> property substitutions in profiles, right?

*-tests artifacts in maven are somewhat special animals and can't be dependent
upon in the common sense. This actually was a reason that BigTop has chosen to
make/use regular binary jar artifacts and use a name designator for their
test-related nature.

With regards,
  Cos