Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> [DISCUSSION] How we gonna test apache builds on real HW


Copy link to this message
-
[DISCUSSION] How we gonna test apache builds on real HW
Folks,

This email is essentially an attempt to work out a community consensus
for a process around testing of Hadoop bits on real hardware.

Here are few pre-requisites:
 - bits are produced by Jenkins builds (e.g.
https://builds.apache.org/hudson/job/Hadoop-22-Build)
 - bits are deployable (e.g. a tar ball or else comprised from all
necessary jars, .so, driver scripts, etc.) immediately and require
only minor configuration changes to run them on a hardware
 - set of tests (a primitive one for the beginning, more elaborate
later on) to check if a produced build is viable and can do sensible
things when is deployed to a cluster (sort of integration tests, if
you will).

At the moment there's a set of static configs (HADOOP-7278) and tiny
deployment script which downloads, unpacks, installs, starts a cluster
on 4 nodes. Script is driven by
https://builds.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-22-cluster-deploy/
job. Successful deploys trigger a test job
(https://builds.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-22-cluster-test/)
which executes some Hadoop examples at the moment.

It is all very quickly put together to have a working validation job
for 0.22 release and I want to limit the scope of this discussion how
we can make a better use of this rather than finding a better solution
for the approach. The latter can be done on JIRA, I think.

Here's a number of questions/answers for the on-cluster testing thing:
 - who should receive notifications about failing deployment tests?
Possible answers: specific release's maillist; an RM; a dev. list;
else?
 - what information should be included into notification to help
identify potential issues: a piece of commit logs; nothing at all?
 - what sets of tests to run. Answers: all hadoop examples, terasort,
pipes, compression jobs, etc.
 - how validation failures should be treated? As a blockers (similar
to functional test failures); "good to know but who cares" kinda
things?

I would like to hear your opinions on this topic.
--
  Thanks,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB