Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Bigtop >> mail # dev >> Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?


Copy link to this message
-
Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?
On Fri, Jan 24, 2014 at 03:28PM, Mark Grover wrote:
> Hey Jay,
> Currently we don't patch things in Bigtop. That means when we download and
> include, say Hadoop 2.0.2 in Bigtop 0.8, our maven artifacts for hadoop
> (say hadoop-common.jar) would have the version 2.2.0 - exactly the same
> version as what upstream hadoop released.

In the interest of full disclosure, we do patch certain things - just grep for
sed in the packages source tree. Builds would be one things; we did patch
source on a couple of occasions in the past to deal with some idiosyncrasies
of the official releases.

But yes - we frown upon such occasions ;)

> So, now 2 options exist for bigpetstore in my opinion:
> 1. Pull upstream Hadoop artifacts from maven. You will rely on Apache
> Hadoop artifacts instead of bigtop artifacts. However, since Bigtop doesn't
> patch, java artifacts should be exactly the same from Bigtop as compared to
> Apache Hadoop.
> 2. Pull Bigtop artifacts for maven. For this, we will obviously need Bigtop
> to a) start updating pom files with its own versioning scheme b) Upload
> them to maven central or equivalent.
>
> As you can see option #2 is a fairly non-trivial overhead for Bigtop but I
> would love to hear if you prefer one of the two options and if so why.

I think what would be most helpful from the ease of development in the stack -
and I have stepped on it more than a few times myself - is to be able to do
maven install for the _whole_ project from the top level pom. As of right now,
one needs to do a bit of rain dance in order to get all the bits in place. And
that's quite annoying apparently. I guess that'd be the next thing to me to
look into it.

Just occurred to me, that if package driving Makefile is replaced with Gradle
that will give a way better consistency of all the parts of the Bigtop
environment and stack. Perhaps, it is my severely under-slept brain is talking
now ;)

Cos

> On Thu, Jan 23, 2014 at 2:25 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:
>
> > Well, on bigpetstore which I'm writing to target bigtop, I have trouble
> > setting up a maven repo with the right dependencies because there are many
> > ecosystem projects.
> >
> > I want to know that my dev environment matches the cluster deployment
> > environment.
> >
> > To do that id like it if there was a maven archetype I could use to build
> > a stub project.  I'm actually planning to make bigpetstore into this
> > archetype app.... So in order to do that I need a systematic way to define
> > the pulled in maven dependencies.
> >
> > Makes sense?
> >
> > > On Jan 23, 2014, at 5:04 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
> > >
> > > Jay,
> > >
> > > I don't think we've been doing bigtop publishing like ever. All use
> > cases that
> > > we had were clearly covered by simple local installation of the
> > artifacts.
> > >
> > > I think it might be a good time to start doing the publishing of the
> > official
> > > Bigtop artifacts to mavencentral.
> > >
> > > Could you get a bit deeper into this whole idea of the applications? I am
> > > interested to hear more ;)!
> > >
> > > Thanks,
> > >  Cos
> > >
> > >> On Thu, Jan 23, 2014 at 03:46PM, Jay Vyas wrote:
> > >> Hi bigtop.
> > >>
> > >> If I want to build an app against bigtop.... Is there a maven repo I can
> > >> point to that has all the version stuff figured out?
> > >>
> > >> As we all know: it's hard to build hadoop apps and test them in a dev
> > >> environment that matches the classpath of cluster environment... I think
> > >> bigtop as a distribution might be able to support something like a
> > >> maven-archetype for bigtop based app development.?
> >

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB