Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # dev - RFC: Major HCatalog refactoring


Copy link to this message
-
Re: RFC: Major HCatalog refactoring
Edward Capriolo 2013-08-31, 14:35
By coverage do you mean to say that:

> Thus, the published HCatalog JARs will contain both packages and the unit
> tests will cover both versions of the API.

We are going to double the time of unit tests for this module?
On Fri, Aug 30, 2013 at 8:41 PM, Eugene Koifman <[EMAIL PROTECTED]>wrote:

> This will change every file under hcatalog so it has to happen before the
> branching.  Most likely at the beginning of next week.
>
> Thanks
>
>
> On Wed, Aug 28, 2013 at 5:24 PM, Eugene Koifman <[EMAIL PROTECTED]
> >wrote:
>
> > Hi,
> >
> >
> > Here is the plan for refactoring HCatalog as was agreed to when it was
> > merged into Hive during.  HIVE-4869 is the umbrella bug for this work.
>  The
> > changes are complex and touch every single file under hcatalog.  Please
> > comment.
> >
> > When HCatalog project was merged into Hive on 0.11 several integration
> > items did not make the 0.11 deadline.  It was agreed to finish them in
> 0.12
> > release.  Specifically:
> >
> > 1. HIVE-4895 - change package name from org.apache.hcatalog to
> > org.apache.hive.hcatalog
> >
> > 2. HIVE-4896 - create binary backwards compatibility layer for hcat users
> > upgrading from 0.11 to 0.12
> >
> > For item 1, we’ll just move every file under org.apache.hcatalog to
> > org.apache.hive.hcatalog and update all “package” and “import” statement
> as
> > well as all hcat/webhcat scripts.  This will include all JUnit tests.
> >
> > Item 2 will ensure that if a user has a M/R program or Pig script, etc.
> > that uses HCatalog public API, their programs will continue to work w/o
> > change with hive 0.12.
> >
> > The proposal is to make the changes that have as little impact on the
> > build system, in part to make upcoming ‘mavenization’ of hive easier, in
> > part to make the changes more manageable.
> >
> >
> >
> > The list of public interfaces (and their transitive closure) for which
> > backwards compat will be provided.
> >
> >    1.
> >
> >    HCatLoader
> >    2.
> >
> >    HCatStorer
> >    3.
> >
> >    HCatInputFormat
> >    4.
> >
> >    HCatOutputFormat
> >    5.
> >
> >    HCatReader
> >    6.
> >
> >    HCatWriter
> >    7.
> >
> >    HCatRecord
> >    8.
> >
> >    HCatSchema
> >
> >
> > To achieve this, 0.11 version of these classes will be added in
> > org.apache.hcatalog package (after item 1 is done).  Each of these
> classes
> > as well as dependencies will be deprecated to make it clear that any new
> > development needs to happen in org.apache.hive.hcatalog.  0.11 version of
> > JUnit tests for hcat will also be brought to trunk and handled the same
> way
> > as mainline code.  A sunset clause will be added to the deprecation
> message.
> >
> > Thus, the published HCatalog JARs will contain both packages and the unit
> > tests will cover both versions of the API.
> >
> > Since these changes are unavoidably disruptive, we’ll need to lock down
> > hcatalog part of hive, check in all existing patches (which are ready,
> i.e.
> > apply/test cleanly and don’t have review comments which need to be
> > addressed) and them make the refactoring changes.
> >
> >
> > Thanks,
> >
> > Eugene
> >
> >
> >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>