Folks,

Given the Hadoop build machines have been offline for a few day, this would be a good time to see if other companies can donate **access**  to some build hardware to diversify our donors.  Currently there are roughly 30 machines donated by Yahoo!

Ideally the hardware is:
* hosted and OS managed by the donor,
* publicly addressable on the internet,
* running Ubuntu or CentOS, and
* sudo access can be given to Apache's Jenkins admins so they can create accounts for committers as needed.

Anyone?

Thanks,
Nige

FWIW, the ASF Infra guys aren't that keen on more machines they won't have
physical control of. There may be some wiggle room on that, but they'll
definitely need sudo and they would much prefer non-single-purpose machines.
Just a heads up from my talks with Infra lately.

A.

On Mon, Aug 1, 2011 at 11:22 PM, Nigel Daley <[EMAIL PROTECTED]> wrote:

> Folks,
>
> Given the Hadoop build machines have been offline for a few day, this would
> be a good time to see if other companies can donate **access**  to some
> build hardware to diversify our donors.  Currently there are roughly 30
> machines donated by Yahoo!
>
> Ideally the hardware is:
> * hosted and OS managed by the donor,
> * publicly addressable on the internet,
> * running Ubuntu or CentOS, and
> * sudo access can be given to Apache's Jenkins admins so they can create
> accounts for committers as needed.
>
> Anyone?
>
> Thanks,
> Nige
>

A couple of questions for Nigel and Andrew:

1. What do you mean by "non-single-purpose machines" ? (I guess, machines
they can use for non-Hadoop projects as well ?)

2. Do these machines have to be physical boxes or virtual machines be okay
too ? (sudo on virtual machines would be easier to sell than on physical
boxes.)

3. What's the process for donations ?

- milind

---
Milind Bhandarkar
Greenplum Labs,
EMC
On 8/2/11 8:07 AM, "Andrew Bayer" <[EMAIL PROTECTED]> wrote:

>FWIW, the ASF Infra guys aren't that keen on more machines they won't have
>physical control of. There may be some wiggle room on that, but they'll
>definitely need sudo and they would much prefer non-single-purpose
>machines.
>Just a heads up from my talks with Infra lately.
>
>A.
>
>On Mon, Aug 1, 2011 at 11:22 PM, Nigel Daley <[EMAIL PROTECTED]> wrote:
>
>> Folks,
>>
>> Given the Hadoop build machines have been offline for a few day, this
>>would
>> be a good time to see if other companies can donate **access**  to some
>> build hardware to diversify our donors.  Currently there are roughly 30
>> machines donated by Yahoo!
>>
>> Ideally the hardware is:
>> * hosted and OS managed by the donor,
>> * publicly addressable on the internet,
>> * running Ubuntu or CentOS, and
>> * sudo access can be given to Apache's Jenkins admins so they can create
>> accounts for committers as needed.
>>
>> Anyone?
>>
>> Thanks,
>> Nige
>>

You should check with ASF Infra for the actual answers - I'm just relaying
my impressions from talking with them.

1) They don't want "targeted donations" - i.e., a donated machine that is
solely for builds of a specific set of projects, rather than one that is
available for builds of any project.

2) I honestly don't know, but as I mentioned, they do prefer to get actual
boxes they can own rather than boxes or VMs living elsewhere.

3) I'm still trying to figure that out myself. =)

A.

On Tue, Aug 2, 2011 at 10:38 AM, <[EMAIL PROTECTED]> wrote:

> A couple of questions for Nigel and Andrew:
>
> 1. What do you mean by "non-single-purpose machines" ? (I guess, machines
> they can use for non-Hadoop projects as well ?)
>
> 2. Do these machines have to be physical boxes or virtual machines be okay
> too ? (sudo on virtual machines would be easier to sell than on physical
> boxes.)
>
> 3. What's the process for donations ?
>
> - milind
>
> ---
> Milind Bhandarkar
> Greenplum Labs,
> EMC
>
>
>
>
> On 8/2/11 8:07 AM, "Andrew Bayer" <[EMAIL PROTECTED]> wrote:
>
> >FWIW, the ASF Infra guys aren't that keen on more machines they won't have
> >physical control of. There may be some wiggle room on that, but they'll
> >definitely need sudo and they would much prefer non-single-purpose
> >machines.
> >Just a heads up from my talks with Infra lately.
> >
> >A.
> >
> >On Mon, Aug 1, 2011 at 11:22 PM, Nigel Daley <[EMAIL PROTECTED]> wrote:
> >
> >> Folks,
> >>
> >> Given the Hadoop build machines have been offline for a few day, this
> >>would
> >> be a good time to see if other companies can donate **access**  to some
> >> build hardware to diversify our donors.  Currently there are roughly 30
> >> machines donated by Yahoo!
> >>
> >> Ideally the hardware is:
> >> * hosted and OS managed by the donor,
> >> * publicly addressable on the internet,
> >> * running Ubuntu or CentOS, and
> >> * sudo access can be given to Apache's Jenkins admins so they can create
> >> accounts for committers as needed.
> >>
> >> Anyone?
> >>
> >> Thanks,
> >> Nige
> >>
>
>

On Aug 1, 2011, at 11:22 PM, Nigel Daley wrote:
> Ideally the hardware is:
> * hosted and OS managed by the donor,
> * publicly addressable on the internet,
> * running Ubuntu or CentOS, and
> * sudo access can be given to Apache's Jenkins admins so they can create accounts for committers as needed.
>

IMO, it would be better to have some diversity rather than what we already have.  The above can be done on EC2.  

It seems to me that HW diversity is good for test machines, but not for
build machines.  Build machines you want to be rock solid and repeatable.
 Test machines you want to span the range of real-world variability.
IMHO,
--Matt

On Tue, Aug 2, 2011 at 10:56 AM, Allen Wittenauer <[EMAIL PROTECTED]> wrote:

>
> On Aug 1, 2011, at 11:22 PM, Nigel Daley wrote:
> > Ideally the hardware is:
> > * hosted and OS managed by the donor,
> > * publicly addressable on the internet,
> > * running Ubuntu or CentOS, and
> > * sudo access can be given to Apache's Jenkins admins so they can create
> accounts for committers as needed.
> >
>
>         IMO, it would be better to have some diversity rather than what we
> already have.  The above can be done on EC2.

As a build guy by trade, I tend to like the idea of doing builds (with unit
tests) on a relative diversity of platforms. But the underlying hardware
should be consistent across the platforms, so that any variations in results
can be narrowed down to the platform differences rather than hardware
differences.

We're also going to need more platforms than just Ubuntu for Bigtop builds,
which is why I've been chatting with Infra on the question of machine
donations. =)

A.

On Tue, Aug 2, 2011 at 11:20 AM, Matt Foley <[EMAIL PROTECTED]> wrote:

> It seems to me that HW diversity is good for test machines, but not for
> build machines.  Build machines you want to be rock solid and repeatable.
>  Test machines you want to span the range of real-world variability.
> IMHO,
> --Matt
>
> On Tue, Aug 2, 2011 at 10:56 AM, Allen Wittenauer <[EMAIL PROTECTED]> wrote:
>
> >
> > On Aug 1, 2011, at 11:22 PM, Nigel Daley wrote:
> > > Ideally the hardware is:
> > > * hosted and OS managed by the donor,
> > > * publicly addressable on the internet,
> > > * running Ubuntu or CentOS, and
> > > * sudo access can be given to Apache's Jenkins admins so they can
> create
> > accounts for committers as needed.
> > >
> >
> >         IMO, it would be better to have some diversity rather than what
> we
> > already have.  The above can be done on EC2.
>

Milind.

>From about 30 boxes Y! gave to ASF at least (I believe there were more, but at
least that much) 10 of them were used _exclusively_ as Hadoop build/test
machines (e.g. Hudson slaves) and a number of others (about 5 at least) were
used for Hadoop deployment testing _exclusively_. So, I think the claim above
doesn't reflect the real situation with hardware donation or perhaps is
missing some of the details ;).

With that in mind I came across this article goo.gl/EVp5S and now I have a
bright idea! Shall the hardware donation be proportional to the proclaimed
market share (or installation base) of a particular commercial distribution's
vendor?  That'd be only fair I guess... Apache Hadoop is about 65% of the
installation and Y! brought in 30 machines. If a vendor A or C has say 30%
then may be they feel good by donating something like 15 boxes to the cause?

--
  Take care,
  Konstantin (Cos) Boudnik
  2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

  Disclaimer: Opinions expressed in this email are those of the author, and do
  not necessarily represent the views of any company the author might be
  affiliated with at the moment of writing.
On Tue, Aug 02, 2011 at 01:38PM, [EMAIL PROTECTED] wrote:
> A couple of questions for Nigel and Andrew:
>
> 1. What do you mean by "non-single-purpose machines" ? (I guess, machines
> they can use for non-Hadoop projects as well ?)
>
> 2. Do these machines have to be physical boxes or virtual machines be okay
> too ? (sudo on virtual machines would be easier to sell than on physical
> boxes.)
>
> 3. What's the process for donations ?
>
> - milind
>
> ---
> Milind Bhandarkar
> Greenplum Labs,
> EMC
>
>
>
>
> On 8/2/11 8:07 AM, "Andrew Bayer" <[EMAIL PROTECTED]> wrote:
>
> >FWIW, the ASF Infra guys aren't that keen on more machines they won't have
> >physical control of. There may be some wiggle room on that, but they'll
> >definitely need sudo and they would much prefer non-single-purpose
> >machines.
> >Just a heads up from my talks with Infra lately.
> >
> >A.
> >
> >On Mon, Aug 1, 2011 at 11:22 PM, Nigel Daley <[EMAIL PROTECTED]> wrote:
> >
> >> Folks,
> >>
> >> Given the Hadoop build machines have been offline for a few day, this
> >>would
> >> be a good time to see if other companies can donate **access**  to some
> >> build hardware to diversify our donors.  Currently there are roughly 30
> >> machines donated by Yahoo!
> >>
> >> Ideally the hardware is:
> >> * hosted and OS managed by the donor,
> >> * publicly addressable on the internet,
> >> * running Ubuntu or CentOS, and
> >> * sudo access can be given to Apache's Jenkins admins so they can create
> >> accounts for committers as needed.
> >>
> >> Anyone?
> >>
> >> Thanks,
> >> Nige
> >>
>

That's good to know, Kos.

Is there any place I can find the actual usage (in terms of utilization /
memory-cpu-disk usage) of these build machines ?

Based on the Amazon EC2 pricing (http://aws.amazon.com/ec2/pricing/), I
would like to determine which type of instances would be adequate, and how
much yearly budget would be needed.

Since ASF is a 501(c)(3) non-profit, charitable organization, these
donations would be tax deductible for U.S. Based individuals. (Plus for
new AWS members, there is a free tier for 1 year.) Is there build infra in
place that will allow these individually donated ec2 instances to be
utilized ?

- Milind
---
Milind Bhandarkar
Greenplum Labs,
EMC
(Disclaimer: Opinions expressed in this email are those of the author, and
do
  not necessarily represent the views of any organization, past or
present, the author might be affiliated with.)

On 8/2/11 11:39 AM, "Konstantin Boudnik" <[EMAIL PROTECTED]> wrote:

>Milind.
>
>From about 30 boxes Y! gave to ASF at least (I believe there were more,
>but at
>least that much) 10 of them were used _exclusively_ as Hadoop build/test
>machines (e.g. Hudson slaves) and a number of others (about 5 at least)
>were
>used for Hadoop deployment testing _exclusively_. So, I think the claim
>above
>doesn't reflect the real situation with hardware donation or perhaps is
>missing some of the details ;).
>
>With that in mind I came across this article goo.gl/EVp5S and now I have a
>bright idea! Shall the hardware donation be proportional to the proclaimed
>market share (or installation base) of a particular commercial
>distribution's
>vendor?  That'd be only fair I guess... Apache Hadoop is about 65% of the
>installation and Y! brought in 30 machines. If a vendor A or C has say 30%
>then may be they feel good by donating something like 15 boxes to the
>cause?
>
>--
>  Take care,
>  Konstantin (Cos) Boudnik
>  2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>
>  Disclaimer: Opinions expressed in this email are those of the author,
>and do
>  not necessarily represent the views of any company the author might be
>  affiliated with at the moment of writing.
>
>
>On Tue, Aug 02, 2011 at 01:38PM, [EMAIL PROTECTED] wrote:
>> A couple of questions for Nigel and Andrew:
>>
>> 1. What do you mean by "non-single-purpose machines" ? (I guess,
>>machines
>> they can use for non-Hadoop projects as well ?)
>>
>> 2. Do these machines have to be physical boxes or virtual machines be
>>okay
>> too ? (sudo on virtual machines would be easier to sell than on physical
>> boxes.)
>>
>> 3. What's the process for donations ?
>>
>> - milind
>>
>> ---
>> Milind Bhandarkar
>> Greenplum Labs,
>> EMC
>>
>>
>>
>>
>> On 8/2/11 8:07 AM, "Andrew Bayer" <[EMAIL PROTECTED]> wrote:
>>
>> >FWIW, the ASF Infra guys aren't that keen on more machines they won't
>>have
>> >physical control of. There may be some wiggle room on that, but they'll
>> >definitely need sudo and they would much prefer non-single-purpose
>> >machines.
>> >Just a heads up from my talks with Infra lately.
>> >
>> >A.
>> >
>> >On Mon, Aug 1, 2011 at 11:22 PM, Nigel Daley <[EMAIL PROTECTED]> wrote:
>> >
>> >> Folks,
>> >>
>> >> Given the Hadoop build machines have been offline for a few day, this
>> >>would
>> >> be a good time to see if other companies can donate **access**  to
>>some
>> >> build hardware to diversify our donors.  Currently there are roughly
>>30
>> >> machines donated by Yahoo!
>> >>
>> >> Ideally the hardware is:
>> >> * hosted and OS managed by the donor,
>> >> * publicly addressable on the internet,
>> >> * running Ubuntu or CentOS, and
>> >> * sudo access can be given to Apache's Jenkins admins so they can
>>create
>> >> accounts for committers as needed.
>> >>
>> >> Anyone?
>> >>
>> >> Thanks,
>> >> Nige
>> >>
>>
>

As I mentioned before, I believe that ASF Infra explicitly does *not* want
donated EC2 instances, but again, check with them to be sure and as to what
the donation process is regardless.

A.

On Tue, Aug 2, 2011 at 12:01 PM, <[EMAIL PROTECTED]> wrote:

> That's good to know, Kos.
>
> Is there any place I can find the actual usage (in terms of utilization /
> memory-cpu-disk usage) of these build machines ?
>
> Based on the Amazon EC2 pricing (http://aws.amazon.com/ec2/pricing/), I
> would like to determine which type of instances would be adequate, and how
> much yearly budget would be needed.
>
> Since ASF is a 501(c)(3) non-profit, charitable organization, these
> donations would be tax deductible for U.S. Based individuals. (Plus for
> new AWS members, there is a free tier for 1 year.) Is there build infra in
> place that will allow these individually donated ec2 instances to be
> utilized ?
>
> - Milind
> ---
> Milind Bhandarkar
> Greenplum Labs,
> EMC
>
>
> (Disclaimer: Opinions expressed in this email are those of the author, and
> do
>  not necessarily represent the views of any organization, past or
> present, the author might be affiliated with.)
>
>
>
>
>
> On 8/2/11 11:39 AM, "Konstantin Boudnik" <[EMAIL PROTECTED]> wrote:
>
> >Milind.
> >
> >From about 30 boxes Y! gave to ASF at least (I believe there were more,
> >but at
> >least that much) 10 of them were used _exclusively_ as Hadoop build/test
> >machines (e.g. Hudson slaves) and a number of others (about 5 at least)
> >were
> >used for Hadoop deployment testing _exclusively_. So, I think the claim
> >above
> >doesn't reflect the real situation with hardware donation or perhaps is
> >missing some of the details ;).
> >
> >With that in mind I came across this article goo.gl/EVp5S and now I have
> a
> >bright idea! Shall the hardware donation be proportional to the proclaimed
> >market share (or installation base) of a particular commercial
> >distribution's
> >vendor?  That'd be only fair I guess... Apache Hadoop is about 65% of the
> >installation and Y! brought in 30 machines. If a vendor A or C has say 30%
> >then may be they feel good by donating something like 15 boxes to the
> >cause?
> >
> >--
> >  Take care,
> >  Konstantin (Cos) Boudnik
> >  2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >
> >  Disclaimer: Opinions expressed in this email are those of the author,
> >and do
> >  not necessarily represent the views of any company the author might be
> >  affiliated with at the moment of writing.
> >
> >
> >On Tue, Aug 02, 2011 at 01:38PM, [EMAIL PROTECTED] wrote:
> >> A couple of questions for Nigel and Andrew:
> >>
> >> 1. What do you mean by "non-single-purpose machines" ? (I guess,
> >>machines
> >> they can use for non-Hadoop projects as well ?)
> >>
> >> 2. Do these machines have to be physical boxes or virtual machines be
> >>okay
> >> too ? (sudo on virtual machines would be easier to sell than on physical
> >> boxes.)
> >>
> >> 3. What's the process for donations ?
> >>
> >> - milind
> >>
> >> ---
> >> Milind Bhandarkar
> >> Greenplum Labs,
> >> EMC
> >>
> >>
> >>
> >>
> >> On 8/2/11 8:07 AM, "Andrew Bayer" <[EMAIL PROTECTED]> wrote:
> >>
> >> >FWIW, the ASF Infra guys aren't that keen on more machines they won't
> >>have
> >> >physical control of. There may be some wiggle room on that, but they'll
> >> >definitely need sudo and they would much prefer non-single-purpose
> >> >machines.
> >> >Just a heads up from my talks with Infra lately.
> >> >
> >> >A.
> >> >
> >> >On Mon, Aug 1, 2011 at 11:22 PM, Nigel Daley <[EMAIL PROTECTED]> wrote:
> >> >
> >> >> Folks,
> >> >>
> >> >> Given the Hadoop build machines have been offline for a few day, this
> >> >>would
> >> >> be a good time to see if other companies can donate **access**  to
> >>some
> >> >> build hardware to diversify our donors.  Currently there are roughly
> >>30
> >> >> machines donated by Yahoo!
> >> >>
> >> >> Ideally the hardware is:
> >> >> * hosted and OS managed by the donor,

Yes, donating hardware to ASF is tricky for all the reasons you've outlined, not the least of which you can't donate to a specific project.  That's why I emphasized **access** to hardware in my request.

Background of current ASF build machines:
1) Y! officially donated 4 machines to ASF, hosting them in a Y! data center.  ASF allocated 2 to Jenkins which helped Hadoop a little bit.  These are minerva and vesta, aka ubuntu1 and ubuntu2 slaves.  The other 2 were allocated to a build bot setup which didn't help Hadoop.
2) A year or so later Y! put another 30 machines in the same rack as the 4 donated ASF machines, but didn't actually donate them to the ASF.   When I asked ASF Infra folks, no one had problems hooking up non ASF hardware to the ASF Jenkins.  Thus from these 30 machines, hadoop1 thru hadoop9 are now Jenkins slaves, plus a bunch of other nodes not connected to Jenkins.

So, donating **access** to hardware for Hadoop builds has been a relatively successful model over the past couple years.  Again, another open to donating **access** to hardware for Hadoop builds?

Cheers,
Nige

On Aug 2, 2011, at 12:08 PM, Andrew Bayer wrote:

> As I mentioned before, I believe that ASF Infra explicitly does *not* want
> donated EC2 instances, but again, check with them to be sure and as to what
> the donation process is regardless.
>
> A.
>
> On Tue, Aug 2, 2011 at 12:01 PM, <[EMAIL PROTECTED]> wrote:
>
>> That's good to know, Kos.
>>
>> Is there any place I can find the actual usage (in terms of utilization /
>> memory-cpu-disk usage) of these build machines ?
>>
>> Based on the Amazon EC2 pricing (http://aws.amazon.com/ec2/pricing/), I
>> would like to determine which type of instances would be adequate, and how
>> much yearly budget would be needed.
>>
>> Since ASF is a 501(c)(3) non-profit, charitable organization, these
>> donations would be tax deductible for U.S. Based individuals. (Plus for
>> new AWS members, there is a free tier for 1 year.) Is there build infra in
>> place that will allow these individually donated ec2 instances to be
>> utilized ?
>>
>> - Milind
>> ---
>> Milind Bhandarkar
>> Greenplum Labs,
>> EMC
>>
>>
>> (Disclaimer: Opinions expressed in this email are those of the author, and
>> do
>> not necessarily represent the views of any organization, past or
>> present, the author might be affiliated with.)
>>
>>
>>
>>
>>
>> On 8/2/11 11:39 AM, "Konstantin Boudnik" <[EMAIL PROTECTED]> wrote:
>>
>>> Milind.
>>>
>>> From about 30 boxes Y! gave to ASF at least (I believe there were more,
>>> but at
>>> least that much) 10 of them were used _exclusively_ as Hadoop build/test
>>> machines (e.g. Hudson slaves) and a number of others (about 5 at least)
>>> were
>>> used for Hadoop deployment testing _exclusively_. So, I think the claim
>>> above
>>> doesn't reflect the real situation with hardware donation or perhaps is
>>> missing some of the details ;).
>>>
>>> With that in mind I came across this article goo.gl/EVp5S and now I have
>> a
>>> bright idea! Shall the hardware donation be proportional to the proclaimed
>>> market share (or installation base) of a particular commercial
>>> distribution's
>>> vendor?  That'd be only fair I guess... Apache Hadoop is about 65% of the
>>> installation and Y! brought in 30 machines. If a vendor A or C has say 30%
>>> then may be they feel good by donating something like 15 boxes to the
>>> cause?
>>>
>>> --
>>> Take care,
>>> Konstantin (Cos) Boudnik
>>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>>
>>> Disclaimer: Opinions expressed in this email are those of the author,
>>> and do
>>> not necessarily represent the views of any company the author might be
>>> affiliated with at the moment of writing.
>>>
>>>
>>> On Tue, Aug 02, 2011 at 01:38PM, [EMAIL PROTECTED] wrote:
>>>> A couple of questions for Nigel and Andrew:
>>>>
>>>> 1. What do you mean by "non-single-purpose machines" ? (I guess,
>>>> machines
>>>> they can use for non-Hadoop projects as well ?)