Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - Hadoop build slaves software


Copy link to this message
-
Re: Hadoop build slaves software
Rajiv Chittajallu 2013-01-04, 19:23
No, there is no CM on these machines. Whats required is mentioned in
the d-i or post scripts.

Since the conversation about build slaves came up, is there a
requirement for having ubuntu. I would prefer to install new machines
with centos. (d-i is a pta. Sorry, I don't want to start a debate here).

-rajive

Todd Lipcon wrote on 01/04/13 at 11:15:34 -0800:
>I've always liked puppet for distributing config files, but always
>though it kind of silly for distributing big binaries like toolchains.
>Seems just as easy to just make a 15-line shell script to wget, tar
>xzf, and make install.
>
>Definitely agree puppet makes sense for ensuring the right deb
>packages are installed on the hosts, though. Are the machines already
>managed by puppet for that?
>
>-Todd
>
>On Fri, Jan 4, 2013 at 11:10 AM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
>> Do I hear puppet? :)
>>
>> Cos
>>
>> On Fri, Jan 04, 2013 at 11:08AM, Todd Lipcon wrote:
>>> I agree -- I'd like to see us have a shell script of some sort which,
>>> given a prefix, downloads and installs the needed toolchain
>>> dependencies.
>>>
>>> We could then download that script onto the build machines and install
>>> into something like /opt/hadoop-toolchain/
>>> AFAIK the only real dependencies we have where the Ubuntu packages are
>>> too old are protoc and maven, so shouldnt be too tough.
>>>
>>> -Todd
>>>
>>> On Fri, Jan 4, 2013 at 10:59 AM, Rajiv Chittajallu <[EMAIL PROTECTED]> wrote:
>>> > asf008 has been up for a while. It was probably just added as a slave.
>>> >
>>> > All the dependencies should probably be installed in a build_prefix, to
>>> > avoid conflict to OS specific packages and allows multiple projects to
>>> > build on the same machines. This is an better alternative to
>>> > provisioning vms for unique builds.
>>> >
>>> > -rajive
>>> >
>>> > Giridharan Kesavan wrote on 01/04/13 at 09:31:55 -0800:
>>> >>   Im on it
>>> >>
>>> >>   -Giri
>>> >>
>>> >>   On Thu, Jan 3, 2013 at 11:24 PM, Todd Lipcon <[1][EMAIL PROTECTED]> wrote:
>>> >>
>>> >>     Hey folks,
>>> >>
>>> >>     It looks like hadoop8 has recently come back online as a build slave,
>>> >>     but is failing all the builds because it has an ancient version of
>>> >>     protobuf (2.2.0):
>>> >>     todd@asf008:~$ protoc  --version
>>> >>     libprotoc 2.2.0
>>> >>
>>> >>     In contrast, other slaves have 2.4.1:
>>> >>     todd@asf001:~$ protoc --version
>>> >>     libprotoc 2.4.1
>>> >>
>>> >>     asf001 has the newer protoc in /usr/local/bin but asf008 does not.
>>> >>     Does anyone know how software is meant to be deployed on these build
>>> >>     slaves? I'm happy to download and install protobuf 2.4.1 into
>>> >>     /usr/local on asf008 if manual installation is the name of the game,
>>> >>     but it seems like we should be doing something a little more
>>> >>     reproducible than one-off builds by rando developers to manage our
>>> >>     toolchain on the Jenkins slaves.
>>> >>     -Todd
>>> >>     --
>>> >>     Todd Lipcon
>>> >>     Software Engineer, Cloudera
>>> >>
>>> >>References
>>> >>
>>> >>   Visible links
>>> >>   1. mailto:[EMAIL PROTECTED]
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>
>
>
>--
>Todd Lipcon
>Software Engineer, Cloudera
>