Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Bigtop >> mail # dev >> Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?


+
Konstantin Boudnik 2014-01-23, 22:05
+
Jay Vyas 2014-01-23, 22:25
Copy link to this message
-
Re: [DISCUSSION] Bigtop based apps: how to get maven deps right?
Hey Jay,
Currently we don't patch things in Bigtop. That means when we download and
include, say Hadoop 2.0.2 in Bigtop 0.8, our maven artifacts for hadoop
(say hadoop-common.jar) would have the version 2.2.0 - exactly the same
version as what upstream hadoop released.

So, now 2 options exist for bigpetstore in my opinion:
1. Pull upstream Hadoop artifacts from maven. You will rely on Apache
Hadoop artifacts instead of bigtop artifacts. However, since Bigtop doesn't
patch, java artifacts should be exactly the same from Bigtop as compared to
Apache Hadoop.
2. Pull Bigtop artifacts for maven. For this, we will obviously need Bigtop
to a) start updating pom files with its own versioning scheme b) Upload
them to maven central or equivalent.

As you can see option #2 is a fairly non-trivial overhead for Bigtop but I
would love to hear if you prefer one of the two options and if so why.

Thanks!
Mark
On Thu, Jan 23, 2014 at 2:25 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:

> Well, on bigpetstore which I'm writing to target bigtop, I have trouble
> setting up a maven repo with the right dependencies because there are many
> ecosystem projects.
>
> I want to know that my dev environment matches the cluster deployment
> environment.
>
> To do that id like it if there was a maven archetype I could use to build
> a stub project.  I'm actually planning to make bigpetstore into this
> archetype app.... So in order to do that I need a systematic way to define
> the pulled in maven dependencies.
>
> Makes sense?
>
> > On Jan 23, 2014, at 5:04 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
> >
> > Jay,
> >
> > I don't think we've been doing bigtop publishing like ever. All use
> cases that
> > we had were clearly covered by simple local installation of the
> artifacts.
> >
> > I think it might be a good time to start doing the publishing of the
> official
> > Bigtop artifacts to mavencentral.
> >
> > Could you get a bit deeper into this whole idea of the applications? I am
> > interested to hear more ;)!
> >
> > Thanks,
> >  Cos
> >
> >> On Thu, Jan 23, 2014 at 03:46PM, Jay Vyas wrote:
> >> Hi bigtop.
> >>
> >> If I want to build an app against bigtop.... Is there a maven repo I can
> >> point to that has all the version stuff figured out?
> >>
> >> As we all know: it's hard to build hadoop apps and test them in a dev
> >> environment that matches the classpath of cluster environment... I think
> >> bigtop as a distribution might be able to support something like a
> >> maven-archetype for bigtop based app development.?
>

 
+
Roman Shaposhnik 2014-01-26, 07:51
+
Mark Grover 2014-01-27, 05:25
+
Jay Vyas 2014-01-27, 13:44
+
Jay Vyas 2014-01-27, 20:08
+
Roman Shaposhnik 2014-01-28, 21:41
+
Konstantin Boudnik 2014-01-28, 17:03
+
Jay Vyas 2014-01-28, 17:06
+
Roman Shaposhnik 2014-01-28, 21:43
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB