Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> RFC: Major HCatalog refactoring


Copy link to this message
-
Re: RFC: Major HCatalog refactoring
Will these be new Java class files or new test methods to existing
classes?  I am just curious as to how this will play into the
distributed testing framework.

On Sat, Aug 31, 2013 at 10:19 AM, Eugene Koifman
<[EMAIL PROTECTED]> wrote:
> not quite double but close  (on my Mac that means it will go up from 35
> minutes to 55-60) so in greater scheme of things it should be negligible
>
>
>
> On Sat, Aug 31, 2013 at 7:35 AM, Edward Capriolo <[EMAIL PROTECTED]>wrote:
>
>> By coverage do you mean to say that:
>>
>> > Thus, the published HCatalog JARs will contain both packages and the unit
>> > tests will cover both versions of the API.
>>
>> We are going to double the time of unit tests for this module?
>>
>>
>> On Fri, Aug 30, 2013 at 8:41 PM, Eugene Koifman <[EMAIL PROTECTED]
>> >wrote:
>>
>> > This will change every file under hcatalog so it has to happen before the
>> > branching.  Most likely at the beginning of next week.
>> >
>> > Thanks
>> >
>> >
>> > On Wed, Aug 28, 2013 at 5:24 PM, Eugene Koifman <
>> [EMAIL PROTECTED]
>> > >wrote:
>> >
>> > > Hi,
>> > >
>> > >
>> > > Here is the plan for refactoring HCatalog as was agreed to when it was
>> > > merged into Hive during.  HIVE-4869 is the umbrella bug for this work.
>> >  The
>> > > changes are complex and touch every single file under hcatalog.  Please
>> > > comment.
>> > >
>> > > When HCatalog project was merged into Hive on 0.11 several integration
>> > > items did not make the 0.11 deadline.  It was agreed to finish them in
>> > 0.12
>> > > release.  Specifically:
>> > >
>> > > 1. HIVE-4895 - change package name from org.apache.hcatalog to
>> > > org.apache.hive.hcatalog
>> > >
>> > > 2. HIVE-4896 - create binary backwards compatibility layer for hcat
>> users
>> > > upgrading from 0.11 to 0.12
>> > >
>> > > For item 1, we’ll just move every file under org.apache.hcatalog to
>> > > org.apache.hive.hcatalog and update all “package” and “import”
>> statement
>> > as
>> > > well as all hcat/webhcat scripts.  This will include all JUnit tests.
>> > >
>> > > Item 2 will ensure that if a user has a M/R program or Pig script, etc.
>> > > that uses HCatalog public API, their programs will continue to work w/o
>> > > change with hive 0.12.
>> > >
>> > > The proposal is to make the changes that have as little impact on the
>> > > build system, in part to make upcoming ‘mavenization’ of hive easier,
>> in
>> > > part to make the changes more manageable.
>> > >
>> > >
>> > >
>> > > The list of public interfaces (and their transitive closure) for which
>> > > backwards compat will be provided.
>> > >
>> > >    1.
>> > >
>> > >    HCatLoader
>> > >    2.
>> > >
>> > >    HCatStorer
>> > >    3.
>> > >
>> > >    HCatInputFormat
>> > >    4.
>> > >
>> > >    HCatOutputFormat
>> > >    5.
>> > >
>> > >    HCatReader
>> > >    6.
>> > >
>> > >    HCatWriter
>> > >    7.
>> > >
>> > >    HCatRecord
>> > >    8.
>> > >
>> > >    HCatSchema
>> > >
>> > >
>> > > To achieve this, 0.11 version of these classes will be added in
>> > > org.apache.hcatalog package (after item 1 is done).  Each of these
>> > classes
>> > > as well as dependencies will be deprecated to make it clear that any
>> new
>> > > development needs to happen in org.apache.hive.hcatalog.  0.11 version
>> of
>> > > JUnit tests for hcat will also be brought to trunk and handled the same
>> > way
>> > > as mainline code.  A sunset clause will be added to the deprecation
>> > message.
>> > >
>> > > Thus, the published HCatalog JARs will contain both packages and the
>> unit
>> > > tests will cover both versions of the API.
>> > >
>> > > Since these changes are unavoidably disruptive, we’ll need to lock down
>> > > hcatalog part of hive, check in all existing patches (which are ready,
>> > i.e.
>> > > apply/test cleanly and don’t have review comments which need to be
>> > > addressed) and them make the refactoring changes.
>> > >
>> > >
>> > > Thanks,
>> > >
>> > > Eugene
>> > >
>> > >
>> > >
>> > >
>> >

Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org