Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Bigtop >> mail # dev >> Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?

Copy link to this message
Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?
Thanks for all this attention to my issue.... I'm trying to follow the debate.

My first question:

1) Can you clarify what is meant by "we drive different dependencies off of our BOM the artifacts end up being different anyway :-("?

Sorry ... Still learning the bigtop vernacular. :).

2) Now in the interim. My current thought is that pig,hive, etc aren't ever gauranteed to have non conflicting classpaths. And thus any hadoop app which attempts to integrate them into a single build will have to use stuff like submodules or profiles.

> On Jan 27, 2014, at 12:24 AM, Mark Grover <[EMAIL PROTECTED]> wrote:
>> On Sat, Jan 25, 2014 at 11:51 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
>>> On Fri, Jan 24, 2014 at 3:28 PM, Mark Grover <[EMAIL PROTECTED]> wrote:
>>> Hey Jay,
>>> Currently we don't patch things in Bigtop. That means when we download
>> and
>>> include, say Hadoop 2.0.2 in Bigtop 0.8, our maven artifacts for hadoop
>>> (say hadoop-common.jar) would have the version 2.2.0 - exactly the same
>>> version as what upstream hadoop released.
>> It is true that we stay away from patching the source, but since we
>> drive different dependencies off of our BOM the artifacts end
>> up being different anyway :-(
>> IOW, an hbase jar from bigtop is NOT the same as published by
>> upstream hbase. In fact it is kind of incompatible. Same goes for
>> most of the other jars with non-trivial dependency chains.
> Ah, good point, Roman. Thanks for correcting me.
>>> So, now 2 options exist for bigpetstore in my opinion:
>>> 1. Pull upstream Hadoop artifacts from maven. You will rely on Apache
>>> Hadoop artifacts instead of bigtop artifacts. However, since Bigtop
>> doesn't
>>> patch, java artifacts should be exactly the same from Bigtop as compared
>> to
>>> Apache Hadoop.
>>> 2. Pull Bigtop artifacts for maven. For this, we will obviously need
>> Bigtop
>>> to a) start updating pom files with its own versioning scheme b) Upload
>>> them to maven central or equivalent.
>>> As you can see option #2 is a fairly non-trivial overhead for Bigtop but
>> I
>>> would love to hear if you prefer one of the two options and if so why.
>> I think the only real alternative is for us to bite the bullet and start
>> publishing bigtop artifacts.
>> In the ideal world -- we'd be pushing right to Maven central, provided
>> that we're careful about branding the artifacts with an explicit Bigtop
>> version stamp. E.g.
>>   org.apache.hadoop:hadoop-annotations:2.2.0-bigtop_0.8.0
> Yeah, I'd be ok with that. The real question is that would we be changing
> our policy to only patch things like version numbers or are we opening up
> the realm of patching in general? Regardless, I think this would be a
> non-trivial cost and we should consider aiming it for later release (0.9?).
>> Thanks,
>> Roman.
> And, welcome back, Roman!
> Mark