Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Bigtop, mail # user - permanant builds on jenkins

Copy link to this message
Re: permanant builds on jenkins
Konstantin Boudnik 2013-07-07, 21:37
[Moving user@ to Bcc: and cross-posting to dev@ This is a more dev discussion,
let's keep it there]

I think there's a confusion of some sort here. Apache normally doesn't provide
binary releases. A release - in ASF - is a source code. While binaries can be
provided as a convenience artifacts it is most often isn't a case.

Now, a Bigtop release is a fixed BOM file that states what versions of what
components were built and tested to work against each other. If you are
looking for particular system packages - it is very easy to get by
  make rpm|deb
from a particular release tag. I believe we do publish binary packages for
each release, but you should understand that these are stored on s3 that costs
money. Right now, BT build infrastructure is running on the funds donated by
commercial vendors (Cloudera is the only one so far, others are chiming in as
we speak).

Same goes for a VM: if you need a VM with particular release bits in it: you
just check out the tag and run boxgrinder that will create identical VM each
time you run the build. The VM in question isn't meant for development of BT.
This is a mere test-bench for people who want to try the latest release of
Hadoop based stack and have no skills of going through the whole process by

I wish we could keep a copy of VMs for all stable builds, but this is a simple
matter of the costs vs. convenience.

As for #3 below: this is pretty much how our Jenkins infra works. There's a
bunch of slave VMs that are used for package building/installation/testing. If
you have some improvements in mind - feel free to open a JIRA tickets and
contribute a patch/design document/etc. Any help is highly welcome.

On Sun, Jul 07, 2013 at 12:03PM, Jay Vyas wrote:
> Hi bruno:
> 1) I've noticed that https://builds.apache.org/view/A-F/view/Bigtop/ is not
> valid but "https://builds.apache.org/job/Bigtop-trunk/" seems to work.  Is
> this an error on the website?
> 2) http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/ Seems to build
> direct from HEAD.
> Are there actual bigtop releases which aren't on jenkins?  Everything seems
> to be centered around the build server, rather than actual frozen,
> downloadable releases.
> 3) Regarding testing the VMs.  Maybe to start some simple virt-install
> scripts could help confirm that the KVM builds, at least, are working.  If
> we could get them working with static IPs then we could even clone down the
> bigtop source code and run bigtop hadoop tests on the VMs after
> construction as a validation step.
> On Sat, Jul 6, 2013 at 8:58 PM, Bruno MahИ <[EMAIL PROTECTED]> wrote:
> > On 07/06/2013 07:33 AM, Jay Vyas wrote:
> >
> >> Hi bigtop:
> >>
> >> Are there any permanant builds saved on jenkins (for the VM matrix)?
> >>
> >> If not it would be nice to add them for certain known well tested,
> >> working disk images .
> >>
> >> (for context, I'm currently running Mr2 build of the KVM box and it
> >> appears to have some intermittent write issues on the DataNode path, and
> >> also, my namenode appears to really like being in safe mode.  these
> >> could just be due to VM setup though, as im changing some things like
> >> adding static IPs and data node write paths... so nothing to be alarmed
> >> about.)
> >>
> >> --
> >> Jay Vyas
> >> http://jayunit100.blogspot.com
> >>
> >
> > Hi Jay,
> >
> > Could you defined "permanent build" ?
> > I am not sure if this fits your requirement, but jenkins has a link to the
> > latest successful build (ex: http://bigtop01.cloudera.org:**
> > 8080/job/Bigtop-VM-matrix/BR=**master,KIND=kvm,label=**
> > fedora16/lastSuccessfulBuild/**artifact/bigtop-vm-kvm-master.**tar.gz<http://bigtop01.cloudera.org:8080/job/Bigtop-VM-matrix/BR=master,KIND=kvm,label=fedora16/lastSuccessfulBuild/artifact/bigtop-vm-kvm-master.tar.gz>)
> >
> > We do not store convenient artifacts of VMs since no one has asked about
> > it before.
> > So ideally, known well tested working disk images would be the ones from