Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> RFC: Major HCatalog refactoring


Copy link to this message
-
Re: RFC: Major HCatalog refactoring
I would say a main goal of unit and integration testing is to try all code
paths. If a testing framework is truly testing all code paths twice, there
is not much of a win there from a unit/integration tests standpoint. If the
unit tests created more coverage of the code that would be an obvious win.
I have not looked at your patch but from your description it sounds like we
are attempting to test a rename that does not sound like a win to me.

If the current hcatalog tests run in 15 minutes, you make a change and then
the run is 30 minutes. 15 minutes is a nice long coffee break, 30 minutes
is a TV show :)

As for the overall hive build taking 10-15 hours. I know that :) I used to
run them, by hand, on my laptop, because no one would share their build
farm with me. I have heard that Hive consumes the vast majority of the
resources of apache's build farm! I think we need to be good citizens at
apache and attempt to make this better, not worse.

Now that we have pre-commit builds we can work at a reasonable pace. Now
that we have this nice pre-commit farm, I do not want to create a precedent
that now we can go "nuts", and start down the same slippery slope.
On Tue, Sep 3, 2013 at 12:57 PM, Eugene Koifman <[EMAIL PROTECTED]>wrote:

> Current (sequential) run of all hive/hcat unit tests takes 10-15 hours.  Is
> another 20-30 minutes that significant?
>
> I'm generally wary of unit tests that are not run continuously and
> automatically.  It delays the detection of problems and then what was
> probably an obvious fix at the time the change was made becomes a long
> debugging session (often by someone other than whose change broke things).
>  I think this is especially true given how many people are contributing to
> hive.
>
>
>
> On Tue, Sep 3, 2013 at 7:25 AM, Brock Noland <[EMAIL PROTECTED]> wrote:
>
> > OK that should be fine.  Though I would echo Edwards sentiment about
> > adding so much test time. Do these tests have to run each time? Does
> > it make sense to have an test target such as test-all-hcatalog and
> > then have then run them periodically manually, especially before
> > releases?
> >
> > On Mon, Sep 2, 2013 at 10:36 AM, Eugene Koifman
> > <[EMAIL PROTECTED]> wrote:
> > > These will be new (I.e. 0.11 version) test classes which will be in the
> > old
> > > org.apache.hcatalog package.  How does that affect the new framework?
> > >
> > > On Saturday, August 31, 2013, Brock Noland wrote:
> > >
> > >> Will these be new Java class files or new test methods to existing
> > >> classes?  I am just curious as to how this will play into the
> > >> distributed testing framework.
> > >>
> > >> On Sat, Aug 31, 2013 at 10:19 AM, Eugene Koifman
> > >> <[EMAIL PROTECTED]> wrote:
> > >> > not quite double but close  (on my Mac that means it will go up from
> > 35
> > >> > minutes to 55-60) so in greater scheme of things it should be
> > negligible
> > >> >
> > >> >
> > >> >
> > >> > On Sat, Aug 31, 2013 at 7:35 AM, Edward Capriolo <
> > [EMAIL PROTECTED]
> > >> >wrote:
> > >> >
> > >> >> By coverage do you mean to say that:
> > >> >>
> > >> >> > Thus, the published HCatalog JARs will contain both packages and
> > the
> > >> unit
> > >> >> > tests will cover both versions of the API.
> > >> >>
> > >> >> We are going to double the time of unit tests for this module?
> > >> >>
> > >> >>
> > >> >> On Fri, Aug 30, 2013 at 8:41 PM, Eugene Koifman <
> > >> [EMAIL PROTECTED]
> > >> >> >wrote:
> > >> >>
> > >> >> > This will change every file under hcatalog so it has to happen
> > before
> > >> the
> > >> >> > branching.  Most likely at the beginning of next week.
> > >> >> >
> > >> >> > Thanks
> > >> >> >
> > >> >> >
> > >> >> > On Wed, Aug 28, 2013 at 5:24 PM, Eugene Koifman <
> > >> >> [EMAIL PROTECTED]
> > >> >> > >wrote:
> > >> >> >
> > >> >> > > Hi,
> > >> >> > >
> > >> >> > >
> > >> >> > > Here is the plan for refactoring HCatalog as was agreed to when
> > it
> > >> was
> > >> >> > > merged into Hive during.  HIVE-4869 is the umbrella bug for
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB