Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # dev >> [DISCUSS] create a hadoop-build subproject (a follow up on the thread 'introduce Python as build-time...')


+
Alejandro Abdelnur 2012-12-05, 17:24
Copy link to this message
-
Re: [DISCUSS] create a hadoop-build subproject (a follow up on the thread 'introduce Python as build-time...')
I think this is a good idea, for a few different reasons.

* We all know Java, so this code will be readable by all.
* It will benefit the wider community of Maven users, not just our project.
* It gets rid of the shell dependency.  The shell dependency is
problematic because Windows doesn't support most UNIX shell code, and
also because different Linux distributions have different default
shells.  The /bin/dash versus /bin/bash issue has caused problems in
the past.

Is there any way this code could be used in branch-1 to solve some of
the issues there?  I know the protoc code, at least, would be useful
there.

regards,
Colin
On Wed, Dec 5, 2012 at 9:24 AM, Alejandro Abdelnur <[EMAIL PROTECTED]> wrote:
> On the '[VOTE] introduce Python as build-time and run-time dependency for
> Hadoop and throughout Hadoop stack' thread the vote for using Maven plugins
> for the build had an overwhelming acceptance (several +1s, a few 0s and no
> -1s).
>
> Current tasks of the build for being 'pluginized' are:
>
> 1* cmake (HADOOP-8887)
> 2* protoc(HADOOP-9117)
> 3* saveVersion (HADOOP-8924) (but this one has been -1)
>
> I've tested #2 and #3 in Windows an the worked as expected. Regarding #1,
> if cmake is supported (and used for hadoop) in Window, i believe it should
> work with minimal changes.
>
> As I mentioned in HADOOP-8924, "Writing Maven plugins is more complex that
> writing scripts, I don't dispute this. The main motivation for using Maven
> plugins is to keep things in the POM declarative and to hide (if necessary)
> different handlings for different platforms."
>
> In order to enable the use of custom plugins for the Hadoop build, we need
> to have such plugins available in a Maven repo for Maven to download and
> use them. Note that they cannot be part of the same build that is building
> Hadoop itself.
>
> While we could argue this plugins are general purpose and they should be
> developed in a different standalone project, doing this would definitely
> slow down when we can use them in Hadoop.
>
> Because of this, I propose we create a Hadoop subproject 'hadoop-build' to
> host custom Maven plugins (and custom Ant tasks for branch-1) for Hadoop.
>
> Being a sub-project will allow us to do independent releases (from Hadoop
> releases) of plugins  we need for Hadoop build. Plus, use them immediately
> in the Hadoop build.
>
> Eventually, if the Maven community picks up some of these plugins we could
> simply remove them from our hadoop-build project and change Hadoop POMs to
> use the new ones.
>
> Looking forward to hear what others think.
>
> Thanks.
>
> --
> Alejandro
+
Alejandro Abdelnur 2012-12-05, 18:41
+
Doug Cutting 2012-12-08, 00:20
+
Radim Kolar 2012-12-08, 01:31
+
Colin McCabe 2012-12-10, 18:50
+
Colin McCabe 2012-12-10, 19:33
+
Radim Kolar 2012-12-12, 03:16
+
Colin McCabe 2012-12-12, 18:22
+
Alejandro Abdelnur 2012-12-12, 18:20
+
Doug Cutting 2012-12-12, 18:46
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB