Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Bigtop >> mail # dev >> Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?


Copy link to this message
-
Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?
On Fri, Jan 24, 2014 at 03:28PM, Mark Grover wrote:
> Hey Jay,
> Currently we don't patch things in Bigtop. That means when we download and
> include, say Hadoop 2.0.2 in Bigtop 0.8, our maven artifacts for hadoop
> (say hadoop-common.jar) would have the version 2.2.0 - exactly the same
> version as what upstream hadoop released.

In the interest of full disclosure, we do patch certain things - just grep for
sed in the packages source tree. Builds would be one things; we did patch
source on a couple of occasions in the past to deal with some idiosyncrasies
of the official releases.

But yes - we frown upon such occasions ;)

> So, now 2 options exist for bigpetstore in my opinion:
> 1. Pull upstream Hadoop artifacts from maven. You will rely on Apache
> Hadoop artifacts instead of bigtop artifacts. However, since Bigtop doesn't
> patch, java artifacts should be exactly the same from Bigtop as compared to
> Apache Hadoop.
> 2. Pull Bigtop artifacts for maven. For this, we will obviously need Bigtop
> to a) start updating pom files with its own versioning scheme b) Upload
> them to maven central or equivalent.
>
> As you can see option #2 is a fairly non-trivial overhead for Bigtop but I
> would love to hear if you prefer one of the two options and if so why.

I think what would be most helpful from the ease of development in the stack -
and I have stepped on it more than a few times myself - is to be able to do
maven install for the _whole_ project from the top level pom. As of right now,
one needs to do a bit of rain dance in order to get all the bits in place. And
that's quite annoying apparently. I guess that'd be the next thing to me to
look into it.

Just occurred to me, that if package driving Makefile is replaced with Gradle
that will give a way better consistency of all the parts of the Bigtop
environment and stack. Perhaps, it is my severely under-slept brain is talking
now ;)

Cos

> On Thu, Jan 23, 2014 at 2:25 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:
>
> > Well, on bigpetstore which I'm writing to target bigtop, I have trouble
> > setting up a maven repo with the right dependencies because there are many
> > ecosystem projects.
> >
> > I want to know that my dev environment matches the cluster deployment
> > environment.
> >
> > To do that id like it if there was a maven archetype I could use to build
> > a stub project.  I'm actually planning to make bigpetstore into this
> > archetype app.... So in order to do that I need a systematic way to define
> > the pulled in maven dependencies.
> >
> > Makes sense?
> >
> > > On Jan 23, 2014, at 5:04 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
> > >
> > > Jay,
> > >
> > > I don't think we've been doing bigtop publishing like ever. All use
> > cases that
> > > we had were clearly covered by simple local installation of the
> > artifacts.
> > >
> > > I think it might be a good time to start doing the publishing of the
> > official
> > > Bigtop artifacts to mavencentral.
> > >
> > > Could you get a bit deeper into this whole idea of the applications? I am
> > > interested to hear more ;)!
> > >
> > > Thanks,
> > >  Cos
> > >
> > >> On Thu, Jan 23, 2014 at 03:46PM, Jay Vyas wrote:
> > >> Hi bigtop.
> > >>
> > >> If I want to build an app against bigtop.... Is there a maven repo I can
> > >> point to that has all the version stuff figured out?
> > >>
> > >> As we all know: it's hard to build hadoop apps and test them in a dev
> > >> environment that matches the classpath of cluster environment... I think
> > >> bigtop as a distribution might be able to support something like a
> > >> maven-archetype for bigtop based app development.?
> >