Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Bigtop >> mail # dev >> Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?


Copy link to this message
-
Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?
YESSSS FOR GRADLE  .
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
On Tue, Jan 28, 2014 at 12:02 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:

> On Fri, Jan 24, 2014 at 03:28PM, Mark Grover wrote:
> > Hey Jay,
> > Currently we don't patch things in Bigtop. That means when we download
> and
> > include, say Hadoop 2.0.2 in Bigtop 0.8, our maven artifacts for hadoop
> > (say hadoop-common.jar) would have the version 2.2.0 - exactly the same
> > version as what upstream hadoop released.
>
> In the interest of full disclosure, we do patch certain things - just grep
> for
> sed in the packages source tree. Builds would be one things; we did patch
> source on a couple of occasions in the past to deal with some
> idiosyncrasies
> of the official releases.
>
> But yes - we frown upon such occasions ;)
>
> > So, now 2 options exist for bigpetstore in my opinion:
> > 1. Pull upstream Hadoop artifacts from maven. You will rely on Apache
> > Hadoop artifacts instead of bigtop artifacts. However, since Bigtop
> doesn't
> > patch, java artifacts should be exactly the same from Bigtop as compared
> to
> > Apache Hadoop.
> > 2. Pull Bigtop artifacts for maven. For this, we will obviously need
> Bigtop
> > to a) start updating pom files with its own versioning scheme b) Upload
> > them to maven central or equivalent.
> >
> > As you can see option #2 is a fairly non-trivial overhead for Bigtop but
> I
> > would love to hear if you prefer one of the two options and if so why.
>
> I think what would be most helpful from the ease of development in the
> stack -
> and I have stepped on it more than a few times myself - is to be able to do
> maven install for the _whole_ project from the top level pom. As of right
> now,
> one needs to do a bit of rain dance in order to get all the bits in place.
> And
> that's quite annoying apparently. I guess that'd be the next thing to me to
> look into it.
>
> Just occurred to me, that if package driving Makefile is replaced with
> Gradle
> that will give a way better consistency of all the parts of the Bigtop
> environment and stack. Perhaps, it is my severely under-slept brain is
> talking
> now ;)
>
> Cos
>
> > On Thu, Jan 23, 2014 at 2:25 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:
> >
> > > Well, on bigpetstore which I'm writing to target bigtop, I have trouble
> > > setting up a maven repo with the right dependencies because there are
> many
> > > ecosystem projects.
> > >
> > > I want to know that my dev environment matches the cluster deployment
> > > environment.
> > >
> > > To do that id like it if there was a maven archetype I could use to
> build
> > > a stub project.  I'm actually planning to make bigpetstore into this
> > > archetype app.... So in order to do that I need a systematic way to
> define
> > > the pulled in maven dependencies.
> > >
> > > Makes sense?
> > >
> > > > On Jan 23, 2014, at 5:04 PM, Konstantin Boudnik <[EMAIL PROTECTED]>
> wrote:
> > > >
> > > > Jay,
> > > >
> > > > I don't think we've been doing bigtop publishing like ever. All use
> > > cases that
> > > > we had were clearly covered by simple local installation of the
> > > artifacts.
> > > >
> > > > I think it might be a good time to start doing the publishing of the
> > > official
> > > > Bigtop artifacts to mavencentral.
> > > >
> > > > Could you get a bit deeper into this whole idea of the applications?
> I am
> > > > interested to hear more ;)!
> > > >
> > > > Thanks,
> > > >  Cos
> > > >
> > > >> On Thu, Jan 23, 2014 at 03:46PM, Jay Vyas wrote:
> > > >> Hi bigtop.
> > > >>
> > > >> If I want to build an app against bigtop.... Is there a maven repo
> I can
> > > >> point to that has all the version stuff figured out?
> > > >>
> > > >> As we all know: it's hard to build hadoop apps and test them in a
> dev
> > > >> environment that matches the classpath of cluster environment... I
> think
> > > >> bigtop as a distribution might be able to support something like a
> > > >> maven-archetype for bigtop based app development.?

Jay Vyas
http://jayunit100.blogspot.com