Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # dev - RFC: Major HCatalog refactoring


Copy link to this message
-
Re: RFC: Major HCatalog refactoring
Edward Capriolo 2013-09-03, 17:22
I would say a main goal of unit and integration testing is to try all code
paths. If a testing framework is truly testing all code paths twice, there
is not much of a win there from a unit/integration tests standpoint. If the
unit tests created more coverage of the code that would be an obvious win.
I have not looked at your patch but from your description it sounds like we
are attempting to test a rename that does not sound like a win to me.

If the current hcatalog tests run in 15 minutes, you make a change and then
the run is 30 minutes. 15 minutes is a nice long coffee break, 30 minutes
is a TV show :)

As for the overall hive build taking 10-15 hours. I know that :) I used to
run them, by hand, on my laptop, because no one would share their build
farm with me. I have heard that Hive consumes the vast majority of the
resources of apache's build farm! I think we need to be good citizens at
apache and attempt to make this better, not worse.

Now that we have pre-commit builds we can work at a reasonable pace. Now
that we have this nice pre-commit farm, I do not want to create a precedent
that now we can go "nuts", and start down the same slippery slope.
On Tue, Sep 3, 2013 at 12:57 PM, Eugene Koifman <[EMAIL PROTECTED]>wrote:

> Current (sequential) run of all hive/hcat unit tests takes 10-15 hours.  Is
> another 20-30 minutes that significant?
>
> I'm generally wary of unit tests that are not run continuously and
> automatically.  It delays the detection of problems and then what was
> probably an obvious fix at the time the change was made becomes a long
> debugging session (often by someone other than whose change broke things).
>  I think this is especially true given how many people are contributing to
> hive.
>
>
>
> On Tue, Sep 3, 2013 at 7:25 AM, Brock Noland <[EMAIL PROTECTED]> wrote:
>
> > OK that should be fine.  Though I would echo Edwards sentiment about
> > adding so much test time. Do these tests have to run each time? Does
> > it make sense to have an test target such as test-all-hcatalog and
> > then have then run them periodically manually, especially before
> > releases?
> >
> > On Mon, Sep 2, 2013 at 10:36 AM, Eugene Koifman
> > <[EMAIL PROTECTED]> wrote:
> > > These will be new (I.e. 0.11 version) test classes which will be in the
> > old
> > > org.apache.hcatalog package.  How does that affect the new framework?
> > >
> > > On Saturday, August 31, 2013, Brock Noland wrote:
> > >
> > >> Will these be new Java class files or new test methods to existing
> > >> classes?  I am just curious as to how this will play into the
> > >> distributed testing framework.
> > >>
> > >> On Sat, Aug 31, 2013 at 10:19 AM, Eugene Koifman
> > >> <[EMAIL PROTECTED]> wrote:
> > >> > not quite double but close  (on my Mac that means it will go up from
> > 35
> > >> > minutes to 55-60) so in greater scheme of things it should be
> > negligible
> > >> >
> > >> >
> > >> >
> > >> > On Sat, Aug 31, 2013 at 7:35 AM, Edward Capriolo <
> > [EMAIL PROTECTED]
> > >> >wrote:
> > >> >
> > >> >> By coverage do you mean to say that:
> > >> >>
> > >> >> > Thus, the published HCatalog JARs will contain both packages and
> > the
> > >> unit
> > >> >> > tests will cover both versions of the API.
> > >> >>
> > >> >> We are going to double the time of unit tests for this module?
> > >> >>
> > >> >>
> > >> >> On Fri, Aug 30, 2013 at 8:41 PM, Eugene Koifman <
> > >> [EMAIL PROTECTED]
> > >> >> >wrote:
> > >> >>
> > >> >> > This will change every file under hcatalog so it has to happen
> > before
> > >> the
> > >> >> > branching.  Most likely at the beginning of next week.
> > >> >> >
> > >> >> > Thanks
> > >> >> >
> > >> >> >
> > >> >> > On Wed, Aug 28, 2013 at 5:24 PM, Eugene Koifman <
> > >> >> [EMAIL PROTECTED]
> > >> >> > >wrote:
> > >> >> >
> > >> >> > > Hi,
> > >> >> > >
> > >> >> > >
> > >> >> > > Here is the plan for refactoring HCatalog as was agreed to when
> > it
> > >> was
> > >> >> > > merged into Hive during.  HIVE-4869 is the umbrella bug for