just for some color, this is the type of hacking we are doing (See readme)
to try to get pig apps to build integration tests in maven. its all
dependency related i beleieve.
So any thoughts on this would be awesome
On Mon, Jan 27, 2014 at 8:43 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:
> Thanks for all this attention to my issue.... I'm trying to follow the
> My first question:
> 1) Can you clarify what is meant by "we drive different dependencies off
> of our BOM the artifacts end up being different anyway :-("?
> Sorry ... Still learning the bigtop vernacular. :).
> 2) Now in the interim. My current thought is that pig,hive, etc aren't
> ever gauranteed to have non conflicting classpaths. And thus any hadoop app
> which attempts to integrate them into a single build will have to use stuff
> like submodules or profiles.
> > On Jan 27, 2014, at 12:24 AM, Mark Grover <[EMAIL PROTECTED]> wrote:
> >> On Sat, Jan 25, 2014 at 11:51 PM, Roman Shaposhnik <[EMAIL PROTECTED]>
> >>> On Fri, Jan 24, 2014 at 3:28 PM, Mark Grover <[EMAIL PROTECTED]> wrote:
> >>> Hey Jay,
> >>> Currently we don't patch things in Bigtop. That means when we download
> >> and
> >>> include, say Hadoop 2.0.2 in Bigtop 0.8, our maven artifacts for hadoop
> >>> (say hadoop-common.jar) would have the version 2.2.0 - exactly the same
> >>> version as what upstream hadoop released.
> >> It is true that we stay away from patching the source, but since we
> >> drive different dependencies off of our BOM the artifacts end
> >> up being different anyway :-(
> >> IOW, an hbase jar from bigtop is NOT the same as published by
> >> upstream hbase. In fact it is kind of incompatible. Same goes for
> >> most of the other jars with non-trivial dependency chains.
> > Ah, good point, Roman. Thanks for correcting me.
> >>> So, now 2 options exist for bigpetstore in my opinion:
> >>> 1. Pull upstream Hadoop artifacts from maven. You will rely on Apache
> >>> Hadoop artifacts instead of bigtop artifacts. However, since Bigtop
> >> doesn't
> >>> patch, java artifacts should be exactly the same from Bigtop as
> >> to
> >>> Apache Hadoop.
> >>> 2. Pull Bigtop artifacts for maven. For this, we will obviously need
> >> Bigtop
> >>> to a) start updating pom files with its own versioning scheme b) Upload
> >>> them to maven central or equivalent.
> >>> As you can see option #2 is a fairly non-trivial overhead for Bigtop
> >> I
> >>> would love to hear if you prefer one of the two options and if so why.
> >> I think the only real alternative is for us to bite the bullet and start
> >> publishing bigtop artifacts.
> >> In the ideal world -- we'd be pushing right to Maven central, provided
> >> that we're careful about branding the artifacts with an explicit Bigtop
> >> version stamp. E.g.
> >> org.apache.hadoop:hadoop-annotations:2.2.0-bigtop_0.8.0
> > Yeah, I'd be ok with that. The real question is that would we be changing
> > our policy to only patch things like version numbers or are we opening up
> > the realm of patching in general? Regardless, I think this would be a
> > non-trivial cost and we should consider aiming it for later release
> >> Thanks,
> >> Roman.
> > And, welcome back, Roman!
> > Mark