Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Bigtop >> mail # dev >> Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?

Copy link to this message
Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?
On Sat, Jan 25, 2014 at 11:51 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:

> On Fri, Jan 24, 2014 at 3:28 PM, Mark Grover <[EMAIL PROTECTED]> wrote:
> > Hey Jay,
> > Currently we don't patch things in Bigtop. That means when we download
> and
> > include, say Hadoop 2.0.2 in Bigtop 0.8, our maven artifacts for hadoop
> > (say hadoop-common.jar) would have the version 2.2.0 - exactly the same
> > version as what upstream hadoop released.
> It is true that we stay away from patching the source, but since we
> drive different dependencies off of our BOM the artifacts end
> up being different anyway :-(
> IOW, an hbase jar from bigtop is NOT the same as published by
> upstream hbase. In fact it is kind of incompatible. Same goes for
> most of the other jars with non-trivial dependency chains.

Ah, good point, Roman. Thanks for correcting me.
> > So, now 2 options exist for bigpetstore in my opinion:
> > 1. Pull upstream Hadoop artifacts from maven. You will rely on Apache
> > Hadoop artifacts instead of bigtop artifacts. However, since Bigtop
> doesn't
> > patch, java artifacts should be exactly the same from Bigtop as compared
> to
> > Apache Hadoop.
> > 2. Pull Bigtop artifacts for maven. For this, we will obviously need
> Bigtop
> > to a) start updating pom files with its own versioning scheme b) Upload
> > them to maven central or equivalent.
> >
> > As you can see option #2 is a fairly non-trivial overhead for Bigtop but
> I
> > would love to hear if you prefer one of the two options and if so why.
> I think the only real alternative is for us to bite the bullet and start
> publishing bigtop artifacts.
> In the ideal world -- we'd be pushing right to Maven central, provided
> that we're careful about branding the artifacts with an explicit Bigtop
> version stamp. E.g.
>    org.apache.hadoop:hadoop-annotations:2.2.0-bigtop_0.8.0

Yeah, I'd be ok with that. The real question is that would we be changing
our policy to only patch things like version numbers or are we opening up
the realm of patching in general? Regardless, I think this would be a
non-trivial cost and we should consider aiming it for later release (0.9?).

> Thanks,
> Roman.

And, welcome back, Roman!