Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Bigtop >> mail # dev >> Review Request 23457: Added ability to build certain Bigtop components against SCM.


Copy link to this message
-
Re: Review Request 23457: Added ability to build certain Bigtop components against SCM.

This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23457/

(Updated July 21, 2014, 7:48 p.m.)
Review request for bigtop.
Changes

This is a final patch one method of doing this where I described this in the Description. You would modify bigtop.mk for the specific project you want to pull from SCM.
Repository: bigtop
Description

I've added the ability to specify a branch, tag, or commit to a download location for a Bigtop component. Once you specify a branch in bigtop.mk and edit a few variables for the package, gradle or make will go to github and fetch a .zip of that specified branch and then build a component with that .zip and package it. I included a commented out HBase example that you can follow for other components if you want to build from SCM for those.

How do I use this?
Just take a loot at the HBase Github example and set the relevant variable configurations for your project. For example, if I wanted to build Hive from master, I would have the following config
# Hive
HIVE_NAME=hive
HIVE_RELNOTES_NAME=Apache Hive
HIVE_PKG_NAME=hive
HIVE_BASE_VERSION=0.13.0
HIVE_PKG_VERSION=$(HIVE_BASE_VERSION)
HIVE_GIT_BRANCH=master
HIVE_RELEASE_VERSION=1
HIVE_TARBALL_DST=apache-hive-$(HIVE_BASE_VERSION)-src.tar.gz
HIVE_TARBALL_SRC=$(HIVE_GIT_BRANCH).zip
HIVE_DOWNLOAD_PATH=/hive/archive
HIVE_SITE=$(GITHUB_MIRROR)$(HIVE_DOWNLOAD_PATH)
HIVE_ARCHIVE=$(HIVE_SITE)
$(eval $(call PACKAGE,hive,HIVE))

So the variables we want to configure or add are _GIT_BRANCH, _TARBALL_SRC, _DOWNLOAD_PATH, _SITE, and _ARCHIVE.

I'm putting this review up to see if everyone likes this method. I'm happy to get recommendations about what to change or do better.

FAQ:
Why didn't you just do git checkout or svn checkout for the components?
That's adding a ton of code to both build systems to be able to handle that and requires both SVN and Git as build dependencies. This was a few line patch to just get a project you want from SCM and get the rest of the projects you need from APACHE_MIRROR. Also, it can be really difficult to find the locations of all the scm's repo's for all projects, whereas from what I've seen all Bigtop components are mirrored to Github, whether or not they're svn or git.

Why not just set _BASE_VERSION to _GIT_BRANCH?
_GIT_BRANCH can be an all letter branch like "master", which we then use _BASE_VERSION to set package versions for debs and rpm's. Deb does not accept all letter versions, and it was easier to just force the person testing against SCM to set a relevant version.

Is the middle of the bigtop.mk file the best place to put the example HBase Github config?
I don't think so, but I didn't know a better place to put it really. I could put it in README if people want or somewhere else.
Diffs (updated)

  Makefile 5d9e209
  bigtop.mk 04c1273
  packages.gradle a059ff6

Diff: https://reviews.apache.org/r/23457/diff/
Testing

built hbase with both make and gradle
Thanks,

Julien Eid