This is with regard to the Kubernetes Scheduler Backend and scaling the
process to accept contributions. Given we're moving past upstreaming
changes from our fork, and into getting *new* patches, I wanted to start
this discussion sooner than later. This is more of a post-2.3 question -
not something we're looking to solve right away.

While unit tests are handy, they're not nearly as good at giving us
confidence as a successful run of our integration tests against
single/multi-node k8s clusters. Currently, we have integration testing
setup at https://github.com/apache-spark-on-k8s/spark-integration and it's
running continuously against apache/spark:master in pepperdata-jenkins
<http://spark-k8s-jenkins.pepperdata.org:8080/view/upstream%20spark/> (on
minikube) & k8s-testgrid
<https://k8s-testgrid.appspot.com/sig-big-data#spark-periodic-default-gke> (in
GKE clusters). Now, the question is - how do we make integration-tests part
of the PR author's workflow?

1. Keep the integration tests in the separate repo and require that
contributors run them, add new tests prior to accepting their PRs as a
policy. Given minikube <https://github.com/kubernetes/minikube> is easy to
setup and can run on a single-node, it would certainly be possible.
Friction however, stems from contributors potentially having to modify the
integration test code hosted in that separate repository when
adding/changing functionality in the scheduler backend. Also, it's
certainly going to lead to at least brief inconsistencies between the two
repositories.

2. Alternatively, we check in the integration tests alongside the actual
scheduler backend code. This would work really well and is what we did in
our fork. It would have to be a separate package which would take certain
parameters (like cluster endpoint) and run integration test code against a
local or remote cluster. It would include least some code dealing with
accessing the cluster, reading results from K8s containers, test fixtures,
etc.

I see value in adopting (2), given it's a clearer path for contributors and
lets us keep the two pieces consistent, but it seems uncommon elsewhere.
How do the other backends, i.e. YARN, Mesos and Standalone deal with
accepting patches and ensuring that they do not break existing clusters? Is
there automation employed for this thus far? Would love to get opinions on
(1) v/s (2).

Thanks,
Anirudh
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB