Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Bigtop >> mail # dev >> Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?


Copy link to this message
-
Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?
Thanks for all this attention to my issue.... I'm trying to follow the debate.

My first question:

1) Can you clarify what is meant by "we drive different dependencies off of our BOM the artifacts end up being different anyway :-("?

Sorry ... Still learning the bigtop vernacular. :).

2) Now in the interim. My current thought is that pig,hive, etc aren't ever gauranteed to have non conflicting classpaths. And thus any hadoop app which attempts to integrate them into a single build will have to use stuff like submodules or profiles.

> On Jan 27, 2014, at 12:24 AM, Mark Grover <[EMAIL PROTECTED]> wrote:
>
>> On Sat, Jan 25, 2014 at 11:51 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
>>
>>> On Fri, Jan 24, 2014 at 3:28 PM, Mark Grover <[EMAIL PROTECTED]> wrote:
>>> Hey Jay,
>>> Currently we don't patch things in Bigtop. That means when we download
>> and
>>> include, say Hadoop 2.0.2 in Bigtop 0.8, our maven artifacts for hadoop
>>> (say hadoop-common.jar) would have the version 2.2.0 - exactly the same
>>> version as what upstream hadoop released.
>>
>> It is true that we stay away from patching the source, but since we
>> drive different dependencies off of our BOM the artifacts end
>> up being different anyway :-(
>>
>> IOW, an hbase jar from bigtop is NOT the same as published by
>> upstream hbase. In fact it is kind of incompatible. Same goes for
>> most of the other jars with non-trivial dependency chains.
>
> Ah, good point, Roman. Thanks for correcting me.
>
>
>>
>>> So, now 2 options exist for bigpetstore in my opinion:
>>> 1. Pull upstream Hadoop artifacts from maven. You will rely on Apache
>>> Hadoop artifacts instead of bigtop artifacts. However, since Bigtop
>> doesn't
>>> patch, java artifacts should be exactly the same from Bigtop as compared
>> to
>>> Apache Hadoop.
>>> 2. Pull Bigtop artifacts for maven. For this, we will obviously need
>> Bigtop
>>> to a) start updating pom files with its own versioning scheme b) Upload
>>> them to maven central or equivalent.
>>>
>>> As you can see option #2 is a fairly non-trivial overhead for Bigtop but
>> I
>>> would love to hear if you prefer one of the two options and if so why.
>>
>> I think the only real alternative is for us to bite the bullet and start
>> publishing bigtop artifacts.
>>
>> In the ideal world -- we'd be pushing right to Maven central, provided
>> that we're careful about branding the artifacts with an explicit Bigtop
>> version stamp. E.g.
>>   org.apache.hadoop:hadoop-annotations:2.2.0-bigtop_0.8.0
>
> Yeah, I'd be ok with that. The real question is that would we be changing
> our policy to only patch things like version numbers or are we opening up
> the realm of patching in general? Regardless, I think this would be a
> non-trivial cost and we should consider aiming it for later release (0.9?).
>
>>
>> Thanks,
>> Roman.
>
> And, welcome back, Roman!
> Mark

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB