Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # dev >> RFC: Major HCatalog refactoring


+
Eugene Koifman 2013-08-29, 00:24
+
Eugene Koifman 2013-08-31, 00:41
+
Edward Capriolo 2013-08-31, 14:35
+
Eugene Koifman 2013-08-31, 15:19
+
Edward Capriolo 2013-08-31, 15:59
+
Brock Noland 2013-08-31, 15:30
+
Eugene Koifman 2013-09-02, 15:36
+
Brock Noland 2013-09-03, 14:25
+
Eugene Koifman 2013-09-03, 16:57
+
Edward Capriolo 2013-09-03, 17:22
Copy link to this message
-
Re: RFC: Major HCatalog refactoring
Edward,

"If a testing framework is truly testing all code paths twice, there
is not much of a win there from a unit/integration tests standpoint. If the
unit tests created more coverage of the code that would be an obvious win.
I have not looked at your patch but from your description it sounds like we
are attempting to test a rename that does not sound like a win to me."

Actually this is not what we are testing.  The package name change (as well
as any changes made in 0.12) will be tested by current tests (which will
also change package name).

The goal of bringing 0.11 version of the source (and corresponding tests)
into 0.12 is to ensure that users who use HCatalog from scripts/MR jobs,
etc (e.g. a Pig script: A = LOAD 'tablename' USING
org.apache.hcatalog.pig.HCatLoader();)  will not have to update all the
their scripts/programs when upgrading to 0.12.  Having 0.11 tests in 0.12
branch ensures that this compatibility layer continues to work while HIve
0.12 and later versions are evolving.

On Tue, Sep 3, 2013 at 10:22 AM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> I would say a main goal of unit and integration testing is to try all code
> paths. If a testing framework is truly testing all code paths twice, there
> is not much of a win there from a unit/integration tests standpoint. If the
> unit tests created more coverage of the code that would be an obvious win.
> I have not looked at your patch but from your description it sounds like we
> are attempting to test a rename that does not sound like a win to me.
>
> If the current hcatalog tests run in 15 minutes, you make a change and then
> the run is 30 minutes. 15 minutes is a nice long coffee break, 30 minutes
> is a TV show :)
>
> As for the overall hive build taking 10-15 hours. I know that :) I used to
> run them, by hand, on my laptop, because no one would share their build
> farm with me. I have heard that Hive consumes the vast majority of the
> resources of apache's build farm! I think we need to be good citizens at
> apache and attempt to make this better, not worse.
>
> Now that we have pre-commit builds we can work at a reasonable pace. Now
> that we have this nice pre-commit farm, I do not want to create a precedent
> that now we can go "nuts", and start down the same slippery slope.
>
>
>
>
> On Tue, Sep 3, 2013 at 12:57 PM, Eugene Koifman <[EMAIL PROTECTED]
> >wrote:
>
> > Current (sequential) run of all hive/hcat unit tests takes 10-15 hours.
>  Is
> > another 20-30 minutes that significant?
> >
> > I'm generally wary of unit tests that are not run continuously and
> > automatically.  It delays the detection of problems and then what was
> > probably an obvious fix at the time the change was made becomes a long
> > debugging session (often by someone other than whose change broke
> things).
> >  I think this is especially true given how many people are contributing
> to
> > hive.
> >
> >
> >
> > On Tue, Sep 3, 2013 at 7:25 AM, Brock Noland <[EMAIL PROTECTED]> wrote:
> >
> > > OK that should be fine.  Though I would echo Edwards sentiment about
> > > adding so much test time. Do these tests have to run each time? Does
> > > it make sense to have an test target such as test-all-hcatalog and
> > > then have then run them periodically manually, especially before
> > > releases?
> > >
> > > On Mon, Sep 2, 2013 at 10:36 AM, Eugene Koifman
> > > <[EMAIL PROTECTED]> wrote:
> > > > These will be new (I.e. 0.11 version) test classes which will be in
> the
> > > old
> > > > org.apache.hcatalog package.  How does that affect the new framework?
> > > >
> > > > On Saturday, August 31, 2013, Brock Noland wrote:
> > > >
> > > >> Will these be new Java class files or new test methods to existing
> > > >> classes?  I am just curious as to how this will play into the
> > > >> distributed testing framework.
> > > >>
> > > >> On Sat, Aug 31, 2013 at 10:19 AM, Eugene Koifman
> > > >> <[EMAIL PROTECTED]> wrote:
> > > >> > not quite double but close  (on my Mac that means it will go up

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
+
Thejas Nair 2013-09-03, 17:20
+
Eugene Koifman 2013-09-03, 19:40
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB