-HADOOP-8545 OpenStack Swift patch ready for commit -reviewers needed
We've got the HADOOP-8545<https://issues.apache.org/jira/browse/HADOOP-8545>patch
stable enough that I'd like it checked in and in the 2.1 beta so that
we can see what problems occur in the field.
The patch adds a new module, hadoop-tools/hadoop-openstack, so it doesn't
break anything in hadoop-common/hdfs/yarn, and it includes more
method-by-method tests than any of the non-HDFS filesystem clients have,
including basic scale tests and the corner-case problems that we're going
to have to talk about in terms of FS API spec and tests. In github, there
are some very basic scale tests running Pig jobs against a configurable
amount of data, showing that we can run pig against GB of data.
We've run these tests against rackspace US, rackspace US and hpcloud -the
public Swift services -as well as private deployments of the latest version
of OpenStack. This makes us confident that we've got authentication right,
and as we get consistent outcomes from all clusters, we aren't relying some
cluster-/provider- specific detail.
So: please can people review it! The site documentation (did I mention
we've got that?) lists how to connect to a swift service, and how to test
against them. If anyone does want to run the test themselves, try the docs
and then escalate to us if there are any problems.
Steve Loughran, Dmitry Mezhenskiy (Mirantis) & David Dobbins (Rackspace)