Well, on bigpetstore which I'm writing to target bigtop, I have trouble setting up a maven repo with the right dependencies because there are many ecosystem projects.
I want to know that my dev environment matches the cluster deployment environment.
To do that id like it if there was a maven archetype I could use to build a stub project. I'm actually planning to make bigpetstore into this archetype app.... So in order to do that I need a systematic way to define the pulled in maven dependencies.
Hey Jay, Currently we don't patch things in Bigtop. That means when we download and include, say Hadoop 2.0.2 in Bigtop 0.8, our maven artifacts for hadoop (say hadoop-common.jar) would have the version 2.2.0 - exactly the same version as what upstream hadoop released.
So, now 2 options exist for bigpetstore in my opinion: 1. Pull upstream Hadoop artifacts from maven. You will rely on Apache Hadoop artifacts instead of bigtop artifacts. However, since Bigtop doesn't patch, java artifacts should be exactly the same from Bigtop as compared to Apache Hadoop. 2. Pull Bigtop artifacts for maven. For this, we will obviously need Bigtop to a) start updating pom files with its own versioning scheme b) Upload them to maven central or equivalent.
As you can see option #2 is a fairly non-trivial overhead for Bigtop but I would love to hear if you prefer one of the two options and if so why.
Thanks! Mark On Thu, Jan 23, 2014 at 2:25 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:
On Fri, Jan 24, 2014 at 3:28 PM, Mark Grover <[EMAIL PROTECTED]> wrote:
It is true that we stay away from patching the source, but since we drive different dependencies off of our BOM the artifacts end up being different anyway :-(
IOW, an hbase jar from bigtop is NOT the same as published by upstream hbase. In fact it is kind of incompatible. Same goes for most of the other jars with non-trivial dependency chains. I think the only real alternative is for us to bite the bullet and start publishing bigtop artifacts.
In the ideal world -- we'd be pushing right to Maven central, provided that we're careful about branding the artifacts with an explicit Bigtop version stamp. E.g. org.apache.hadoop:hadoop-annotations:2.2.0-bigtop_0.8.0
On Sat, Jan 25, 2014 at 11:51 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote: Ah, good point, Roman. Thanks for correcting me.
Yeah, I'd be ok with that. The real question is that would we be changing our policy to only patch things like version numbers or are we opening up the realm of patching in general? Regardless, I think this would be a non-trivial cost and we should consider aiming it for later release (0.9?). And, welcome back, Roman! Mark
Thanks for all this attention to my issue.... I'm trying to follow the debate.
My first question:
1) Can you clarify what is meant by "we drive different dependencies off of our BOM the artifacts end up being different anyway :-("?
Sorry ... Still learning the bigtop vernacular. :).
2) Now in the interim. My current thought is that pig,hive, etc aren't ever gauranteed to have non conflicting classpaths. And thus any hadoop app which attempts to integrate them into a single build will have to use stuff like submodules or profiles.
On Fri, Jan 24, 2014 at 03:28PM, Mark Grover wrote:
In the interest of full disclosure, we do patch certain things - just grep for sed in the packages source tree. Builds would be one things; we did patch source on a couple of occasions in the past to deal with some idiosyncrasies of the official releases.
But yes - we frown upon such occasions ;) I think what would be most helpful from the ease of development in the stack - and I have stepped on it more than a few times myself - is to be able to do maven install for the _whole_ project from the top level pom. As of right now, one needs to do a bit of rain dance in order to get all the bits in place. And that's quite annoying apparently. I guess that'd be the next thing to me to look into it.
Just occurred to me, that if package driving Makefile is replaced with Gradle that will give a way better consistency of all the parts of the Bigtop environment and stack. Perhaps, it is my severely under-slept brain is talking now ;)
On Mon, Jan 27, 2014 at 5:43 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:
Basically I am referring to the fact that within Bigtop all component dependencies are just that -- within Bigtop. IOW, Bigtop's HBase build will be forced to build against the version of Hadoop that is part of the same BOM, not the default version of Hadoop that is defined in upstream HBase pom. Since we drive dependencies in this way we are essentially changing the artifacts. This one I am quite sure I follow.
On Tue, Jan 28, 2014 at 9:02 AM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
Hm. That's actually an intriguing possibility.
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext