Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # dev >> RFC: Major HCatalog refactoring


+
Eugene Koifman 2013-08-29, 00:24
+
Eugene Koifman 2013-08-31, 00:41
+
Edward Capriolo 2013-08-31, 14:35
+
Eugene Koifman 2013-08-31, 15:19
+
Edward Capriolo 2013-08-31, 15:59
+
Brock Noland 2013-08-31, 15:30
+
Eugene Koifman 2013-09-02, 15:36
+
Brock Noland 2013-09-03, 14:25
+
Eugene Koifman 2013-09-03, 16:57
+
Edward Capriolo 2013-09-03, 17:22
+
Eugene Koifman 2013-09-03, 18:13
+
Thejas Nair 2013-09-03, 17:20
Copy link to this message
-
Re: RFC: Major HCatalog refactoring
The plan was 2 releases.
On Tue, Sep 3, 2013 at 12:14 PM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> You may have already said this but remind me again. If we go with this
> approach, how long until we retired the duplicated code and insist end
> users use the new name? 1 release?
>
> A similar debate is likely why the hive classes are still packaged as
> org.apache.hadoop.hive, rather then org.apache.hive.
>
>
> On Tue, Sep 3, 2013 at 2:54 PM, Eugene Koifman <[EMAIL PROTECTED]
> >wrote:
>
> > We explored the idea you suggest and given the number of APIs (and their
> > transitive closure) it would would be very difficult and the result would
> > be fragile.  So unfortunately that is not possible.
> >
> > For example, oldpackage.A has a  method foo() that returns oldpackage.B.
> >  You could create
> > newpackage.A extends oldpackage.A {
> >  @Override
> >  newpacage.B foo() {
> >  }
> > }
> >
> > which works because of covariant return type, but the implementation of
> > foo() becomes problematic because it itself uses other classes.
> >
> > On Tue, Sep 3, 2013 at 11:41 AM, Edward Capriolo <[EMAIL PROTECTED]
> > >wrote:
> >
> > > I understand.
> > >
> > > Can we do something like this?
> > >
> > > oldpackage.HCatologLoader extends newpackage.HCatlogloader { }
> > >
> > > If we do something like this we don't need to test both classes, it is
> > safe
> > > to assume they both do the same thing.
> > >
> > > I understand that we do not want users to have to specify a new class
> > name,
> > > but 15 minutes of unit tests around a re-name is overkill.
> > >
> > >
> > > On Tue, Sep 3, 2013 at 2:13 PM, Eugene Koifman <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Edward,
> > > >
> > > > "If a testing framework is truly testing all code paths twice, there
> > > > is not much of a win there from a unit/integration tests standpoint.
> If
> > > the
> > > > unit tests created more coverage of the code that would be an obvious
> > > win.
> > > > I have not looked at your patch but from your description it sounds
> > like
> > > we
> > > > are attempting to test a rename that does not sound like a win to
> me."
> > > >
> > > > Actually this is not what we are testing.  The package name change
> (as
> > > well
> > > > as any changes made in 0.12) will be tested by current tests (which
> > will
> > > > also change package name).
> > > >
> > > > The goal of bringing 0.11 version of the source (and corresponding
> > tests)
> > > > into 0.12 is to ensure that users who use HCatalog from scripts/MR
> > jobs,
> > > > etc (e.g. a Pig script: A = LOAD 'tablename' USING
> > > > org.apache.hcatalog.pig.HCatLoader();)  will not have to update all
> the
> > > > their scripts/programs when upgrading to 0.12.  Having 0.11 tests in
> > 0.12
> > > > branch ensures that this compatibility layer continues to work while
> > HIve
> > > > 0.12 and later versions are evolving.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Sep 3, 2013 at 10:22 AM, Edward Capriolo <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > I would say a main goal of unit and integration testing is to try
> all
> > > > code
> > > > > paths. If a testing framework is truly testing all code paths
> twice,
> > > > there
> > > > > is not much of a win there from a unit/integration tests
> standpoint.
> > If
> > > > the
> > > > > unit tests created more coverage of the code that would be an
> obvious
> > > > win.
> > > > > I have not looked at your patch but from your description it sounds
> > > like
> > > > we
> > > > > are attempting to test a rename that does not sound like a win to
> me.
> > > > >
> > > > > If the current hcatalog tests run in 15 minutes, you make a change
> > and
> > > > then
> > > > > the run is 30 minutes. 15 minutes is a nice long coffee break, 30
> > > minutes
> > > > > is a TV show :)
> > > > >
> > > > > As for the overall hive build taking 10-15 hours. I know that :) I
> > used
> > > > to
> > > > > run them, by hand, on my laptop, because no one would share their

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.