Thanks for the reminder Stack.

I got a little sidetracked after working at 3AM this morning :-)

As Stack has demonstrated through adding unit tests for HBASE-4298, feature
patches would get accepted faster if committers are willing to nurture them.
There is usually considerable amount of work behind feature patches, such
as HBASE-451, HBASE-4120, HBASE-4213, etc from both contributors and
committers.
Such effort tends to last several months. So a little touch up now and then
is healthy for the final integration of the features.

The above is my personal opinion.

On Wed, Nov 2, 2011 at 2:46 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Wed, Nov 2, 2011 at 2:05 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > Todd:
> > May I reserve HBASE-4120 as the top JIRA to be discussed once 0.92 is
> > considered stable ?
> >
>
> What are you trying to reserve Ted?  Developer attention?  I think the
> only way you would be able to do that is if you hire us all.
> St.Ack
>

On Wed, Nov 2, 2011 at 4:40 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> Such effort tends to last several months. So a little touch up now and then
> is healthy for the final integration of the features.
>
> The above is my personal opinion.

My opinion: it's up to those who want a feature to get integrated to
make sure it's integratable. That means:
- fits codebase style
- get agreement on architecture/approach if it's a big patch or
changes/adds APIs
- write sufficient tests
- do testing on a real cluster (at least a few nodes) if it involves
distributed operations

Those who "want a feature" might be just the contributor, just a
committer, or some combination of the above. But if a contributor
shows up with a feature that isn't high priority for any committers, I
don't think any of us have a responsibility to do the above _for_
them. We _do_ have a responsibility to make the above guidelines clear
and apply them equally regardless of who the contributor happens to
be. (eg if Facebook shows up with a patch that doesn't have unit
tests, we should treat them the same as an unknown contributor)

The above isn't in reference to the patch that started this discussion
-- just my opinions on how successful open source works.

I'm probably less idealistic than some other folks -- IMO our time
contribution to HBase isn't charity. We do it because our businesses
rely on it (either directly or indirectly), and thus we should expect
that everyone acts mostly with self-interest. Our self-interest of
course is highly aligned with the success and stability of the
project, which is why this all tends to work out!

-Todd

>
> On Wed, Nov 2, 2011 at 2:46 PM, Stack <[EMAIL PROTECTED]> wrote:
>
>> On Wed, Nov 2, 2011 at 2:05 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> > Todd:
>> > May I reserve HBASE-4120 as the top JIRA to be discussed once 0.92 is
>> > considered stable ?
>> >
>>
>> What are you trying to reserve Ted?  Developer attention?  I think the
>> only way you would be able to do that is if you hire us all.
>> St.Ack
>>
>

--
Todd Lipcon
Software Engineer, Cloudera

I agree with Todd's points.

w.r.t. HBASE-4120 and multi-tenant HBase clusters, allow me to point to
this discussion:

https://issues.apache.org/jira/browse/HBASE-4329?focusedCommentId=13096986&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13096986

I think there are use cases for supporting multi-tenant HBase clusters
where HBASE-4120 provides one avenue.

Cheers

On Wed, Nov 2, 2011 at 4:50 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Wed, Nov 2, 2011 at 4:40 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > Such effort tends to last several months. So a little touch up now and
> then
> > is healthy for the final integration of the features.
> >
> > The above is my personal opinion.
>
> My opinion: it's up to those who want a feature to get integrated to
> make sure it's integratable. That means:
> - fits codebase style
> - get agreement on architecture/approach if it's a big patch or
> changes/adds APIs
> - write sufficient tests
> - do testing on a real cluster (at least a few nodes) if it involves
> distributed operations
>
> Those who "want a feature" might be just the contributor, just a
> committer, or some combination of the above. But if a contributor
> shows up with a feature that isn't high priority for any committers, I
> don't think any of us have a responsibility to do the above _for_
> them. We _do_ have a responsibility to make the above guidelines clear
> and apply them equally regardless of who the contributor happens to
> be. (eg if Facebook shows up with a patch that doesn't have unit
> tests, we should treat them the same as an unknown contributor)
>
> The above isn't in reference to the patch that started this discussion
> -- just my opinions on how successful open source works.
>
> I'm probably less idealistic than some other folks -- IMO our time
> contribution to HBase isn't charity. We do it because our businesses
> rely on it (either directly or indirectly), and thus we should expect
> that everyone acts mostly with self-interest. Our self-interest of
> course is highly aligned with the success and stability of the
> project, which is why this all tends to work out!
>
> -Todd
>
> >
> > On Wed, Nov 2, 2011 at 2:46 PM, Stack <[EMAIL PROTECTED]> wrote:
> >
> >> On Wed, Nov 2, 2011 at 2:05 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >> > Todd:
> >> > May I reserve HBASE-4120 as the top JIRA to be discussed once 0.92 is
> >> > considered stable ?
> >> >
> >>
> >> What are you trying to reserve Ted?  Developer attention?  I think the
> >> only way you would be able to do that is if you hire us all.
> >> St.Ack
> >>
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

I agree with Todd's sentiments on unnecessarily coupling feature priority
with the availability of a patch.  There's patches that we've developed
internally, then threw away because we couldn't tie it to a production use
case or thought a better design might be necessary.  There's also patches
that we've had simmering out for a couple weeks so we could think about it
and cross-talk.  Every new feature adds to the complexity of an
already-complex system.  We should make sure the feature has demand.

I don't think tying the maturity of a patch merely to a metric as simple
as the presence of a unit test.  Unit tests are great, but they're a means
instead of an ends to quality.  I've seen super-buggy code that had a
number of unit tests.  I've seen code that passed the unit test, but the
test assumptions were wrong.  I've seen code that failed a unit test
because the unit test was horribly written.  There's a lot of metrics that
make a feature more palatable.  How are you using this?  Can you give
examples? Is it in production or just something you wanted?  How actively
is this being used?  What sort of scaling is being tested here?  These are
questions that contributors should be available to answer if they submit a
large diff that impacts a critical section of code and will be a necessary
design decision for many future features.  

On Wed, Nov 2, 2011 at 5:12 PM, Nicolas Spiegelberg <[EMAIL PROTECTED]> wrote:
> I don't think tying the maturity of a patch merely to a metric as simple
> as the presence of a unit test.  Unit tests are great, but they're a means
> instead of an ends to quality.  I've seen super-buggy code that had a
> number of unit tests.  I've seen code that passed the unit test, but the
> test assumptions were wrong.  I've seen code that failed a unit test
> because the unit test was horribly written.  There's a lot of metrics that
> make a feature more palatable.  How are you using this?  Can you give
> examples? Is it in production or just something you wanted?  How actively
> is this being used?  What sort of scaling is being tested here?  These are
> questions that contributors should be available to answer if they submit a
> large diff that impacts a critical section of code and will be a necessary
> design decision for many future features.

I phrased this as "write sufficient tests" since I mostly agree with
the above. It's possible to have lots of unit tests which show
nothing, or few tests which convince people that it works. The measure
isn't binary, even though the QA bot will give a "+1 tests included"
for including a single test.

I don't think it's OK to only have manual tests. Manual tests (or
stable production deployments) prove that the feature is stable
_today_ but do not show that the feature is stable tomorrow or the
next week. Unless we had a QA team and a list of text files checked in
with all of the manual tests needed to certify a release, we'd end up
releasing things with good chances of regressions.

Adding a system test suite would do us some good here. Accumulo has a
very nice one which we could work on cloning (though the APIs are
different we can borrow the test scenarios).

-Todd
--
Todd Lipcon
Software Engineer, Cloudera

Todd,

I am curious what you mean here.  How is adding a test suite better than
annotating different tests in the TestNG or Junit 4 style?

On Wed, Nov 2, 2011 at 5:25 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> Adding a system test suite would do us some good here. Accumulo has a
> very nice one which we could work on cloning (though the APIs are
> different we can borrow the test scenarios).
>

On Wed, Nov 2, 2011 at 5:25 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> Adding a system test suite would do us some good here. Accumulo has a
> very nice one which we could work on cloning (though the APIs are
> different we can borrow the test scenarios).

Essentially, that's what I'm doing over here:
    http://bigtop01.cloudera.org:8080/view/Hadoop%200.22/job/Bigtop-trunk-smoketest-22/lastCompletedBuild/testReport/
with the smalish test suite (Todd, you should recognize some of
the tests -- I think you wrote them ;-))

As I mentioned before, I'd love to expand on that. But so far
I haven't heard anybody willing to contribute to:
   https://svn.apache.org/repos/asf/incubator/bigtop/trunk/bigtop-tests/test-artifacts/hbase/

Thanks,
Roman.

On Wed, Nov 2, 2011 at 5:31 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:
> Todd,
>
> I am curious what you mean here.  How is adding a test suite better than
> annotating different tests in the TestNG or Junit 4 style?

By "test suite" I wasn't referring to an implementation style. I'm
referring to a testing style in which the tests would run against a
real live deployed cluster, generate load, run for several
hours/continuously, etc.

https://github.com/apache/accumulo/tree/trunk/test/system are the
tests I referred to from Accumulo.

-Todd

>
> On Wed, Nov 2, 2011 at 5:25 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> Adding a system test suite would do us some good here. Accumulo has a
>> very nice one which we could work on cloning (though the APIs are
>> different we can borrow the test scenarios).
>>
>

--
Todd Lipcon
Software Engineer, Cloudera

On Wed, Nov 2, 2011 at 5:33 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Wed, Nov 2, 2011 at 5:31 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:
> > Todd,
> >
> > I am curious what you mean here.  How is adding a test suite better than
> > annotating different tests in the TestNG or Junit 4 style?
>
> By "test suite" I wasn't referring to an implementation style. I'm
> referring to a testing style in which the tests would run against a
> real live deployed cluster, generate load, run for several
> hours/continuously, etc.
>

Ahh... not a Junit test suite.

The real thing.

On Wed, Nov 2, 2011 at 5:39 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:
>> By "test suite" I wasn't referring to an implementation style. I'm
>> referring to a testing style in which the tests would run against a
>> real live deployed cluster, generate load, run for several
>> hours/continuously, etc.
>>
>
> Ahh... not a Junit test suite.

Nothing says you can't use JUnit/TestNG style of writing testcase to
write integration/system tests. But yes, not the Unit test suite.

Thanks,
Roman.

On Wed, Nov 2, 2011 at 5:31 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
> As I mentioned before, I'd love to expand on that. But so far
> I haven't heard anybody willing to contribute to:
>   https://svn.apache.org/repos/asf/incubator/bigtop/trunk/bigtop-tests/test-artifacts/hbase/
>

I'm volunteering to help out Roman.  A suite of tests that run against
a cluster to verify stuff is basically as good/bad as it has always
been is I think top priority for the project and plugs in nicely with
the work nkeywal is doing refactoring our unit tests.

St.Ack

On Wed, Nov 2, 2011 at 5:12 PM, Nicolas Spiegelberg <[EMAIL PROTECTED]> wrote:
> I agree with Todd's sentiments on unnecessarily coupling feature priority
> with the availability of a patch.  There's patches that we've developed
> internally, then threw away because we couldn't tie it to a production use
> case or thought a better design might be necessary.  There's also patches
> that we've had simmering out for a couple weeks so we could think about it
> and cross-talk.  Every new feature adds to the complexity of an
> already-complex system.  We should make sure the feature has demand.
>

Another factor that I think we should consider when taking on big
features, and I've heard a version of this from you Nicolas, is
whether the feature contributor will be around post commit to fix bugs
that pop up post-commit.

St.Ack

Nicolas:
Your earlier comment on metrics that make a feature more palatable is very
good.

Can you take a look at and see if there is more information you want to
gather ?
https://issues.apache.org/jira/browse/HBASE-4120?focusedCommentId=13143315&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13143315

Stack:
Once a feature goes to production at company X, they would dedicate
personnel on maintaining the feature.

Cheers

On Wed, Nov 2, 2011 at 8:17 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Wed, Nov 2, 2011 at 5:12 PM, Nicolas Spiegelberg <[EMAIL PROTECTED]>
> wrote:
> > I agree with Todd's sentiments on unnecessarily coupling feature priority
> > with the availability of a patch.  There's patches that we've developed
> > internally, then threw away because we couldn't tie it to a production
> use
> > case or thought a better design might be necessary.  There's also patches
> > that we've had simmering out for a couple weeks so we could think about
> it
> > and cross-talk.  Every new feature adds to the complexity of an
> > already-complex system.  We should make sure the feature has demand.
> >
>
> Another factor that I think we should consider when taking on big
> features, and I've heard a version of this from you Nicolas, is
> whether the feature contributor will be around post commit to fix bugs
> that pop up post-commit.
>
> St.Ack
>

> Another factor that I think we should consider when taking on big
> features, and I've heard a version of this from you Nicolas, is
> whether the feature contributor will be around post commit to fix bugs
> that pop up post-commit.

We've been burned by that a few times that I can recall but isn't that par for the course for open source? I'm not sure there is much that we can do there. How would this be determined? Circumstances change. 

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
>From: Stack <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]
>Sent: Wednesday, November 2, 2011 8:17 PM
>Subject: Re: patch maturity and HBase release Was: HBASE-4120 table level priority
>
>On Wed, Nov 2, 2011 at 5:12 PM, Nicolas Spiegelberg <[EMAIL PROTECTED]> wrote:
>> I agree with Todd's sentiments on unnecessarily coupling feature priority
>> with the availability of a patch.  There's patches that we've developed
>> internally, then threw away because we couldn't tie it to a production use
>> case or thought a better design might be necessary.  There's also patches
>> that we've had simmering out for a couple weeks so we could think about it
>> and cross-talk.  Every new feature adds to the complexity of an
>> already-complex system.  We should make sure the feature has demand.
>>
>
>Another factor that I think we should consider when taking on big
>features, and I've heard a version of this from you Nicolas, is
>whether the feature contributor will be around post commit to fix bugs
>that pop up post-commit.
>
>St.Ack
>
>
> 

On Wed, Nov 2, 2011 at 8:11 PM, Stack <[EMAIL PROTECTED]> wrote:
> On Wed, Nov 2, 2011 at 5:31 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
>> As I mentioned before, I'd love to expand on that. But so far
>> I haven't heard anybody willing to contribute to:
>>   https://svn.apache.org/repos/asf/incubator/bigtop/trunk/bigtop-tests/test-artifacts/hbase/
>>
>
> I'm volunteering to help out Roman.  A suite of tests that run against
> a cluster to verify stuff is basically as good/bad as it has always
> been is I think top priority for the project and plugs in nicely with
> the work nkeywal is doing refactoring our unit tests.

I'd love to see that happen! Here's what we have to offer from the
Bigtop side of things:
   1. A place to commit these tests:
          https://svn.apache.org/repos/asf/incubator/bigtop/trunk/bigtop-tests/test-artifacts/hbase/

   2. An [evolving] integration test library/framework called iTest
(based on JUnit/Maven):
          https://svn.apache.org/repos/asf/incubator/bigtop/trunk/bigtop-tests/test-execution/
          https://svn.apache.org/repos/asf/incubator/bigtop/trunk/bigtop-test-framework/

   3. A way to rapidly build a desired Hadoop stack and deploy it to
an on-demand
       cluster:
           https://svn.apache.org/repos/asf/incubator/bigtop/trunk/bigtop-deploy/puppet/

   4. A Jenkins server with enough of slaves in place to tie #1-#3
together and collate the
       test results:
            http://bigtop01.cloudera.org:8080/view/Hadoop%200.22/

Writing tests is not difficult at all (just take a look at existing
ones and you should
see how its done). We also need to improve the DOCs on everything that's
related to iTest, but that hasn't happened yet -- so if you're interested feel
free to email me personally of head over to [EMAIL PROTECTED].

What we're missing in Bigtop is deep domain knowledge of the components
and their integration testing needs. The existing tests are basically for the
features that we cared about for one reason or another (e.g. Compression
codecs, Pig/HBase interop, etc.) and they are not an HBase view of integration
testing needs.  But hey -- its better than nothing! ;-)

Also, a low-hanging-fruit sort of approach that we've taken with Hadoop
and Pig is to try and re-use existing unit tests in the context of Integration
testing. A good example here is TestCLI which runs a whole bunch of
FS operations and compares results to the golden file. This is the type of
test that we can run against a real cluster (baring some minor modifications
that went upstream eventually). Thus if there's a subset of HBase unit
tests that we can reuse in such a context -- that'll be very nice.
Thanks,
Roman.

On Thu, Nov 3, 2011 at 3:56 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
> I'd love to see that happen! Here's what we have to offer from the
> Bigtop side of things:

...

Thanks for the write up Roman. Sounds just right.  Will be by after
the 0.92 RC goes up.
St.Ack