Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> RFC: Major HCatalog refactoring


Copy link to this message
-
Re: RFC: Major HCatalog refactoring
OK that should be fine.  Though I would echo Edwards sentiment about
adding so much test time. Do these tests have to run each time? Does
it make sense to have an test target such as test-all-hcatalog and
then have then run them periodically manually, especially before
releases?

On Mon, Sep 2, 2013 at 10:36 AM, Eugene Koifman
<[EMAIL PROTECTED]> wrote:
> These will be new (I.e. 0.11 version) test classes which will be in the old
> org.apache.hcatalog package.  How does that affect the new framework?
>
> On Saturday, August 31, 2013, Brock Noland wrote:
>
>> Will these be new Java class files or new test methods to existing
>> classes?  I am just curious as to how this will play into the
>> distributed testing framework.
>>
>> On Sat, Aug 31, 2013 at 10:19 AM, Eugene Koifman
>> <[EMAIL PROTECTED]> wrote:
>> > not quite double but close  (on my Mac that means it will go up from 35
>> > minutes to 55-60) so in greater scheme of things it should be negligible
>> >
>> >
>> >
>> > On Sat, Aug 31, 2013 at 7:35 AM, Edward Capriolo <[EMAIL PROTECTED]
>> >wrote:
>> >
>> >> By coverage do you mean to say that:
>> >>
>> >> > Thus, the published HCatalog JARs will contain both packages and the
>> unit
>> >> > tests will cover both versions of the API.
>> >>
>> >> We are going to double the time of unit tests for this module?
>> >>
>> >>
>> >> On Fri, Aug 30, 2013 at 8:41 PM, Eugene Koifman <
>> [EMAIL PROTECTED]
>> >> >wrote:
>> >>
>> >> > This will change every file under hcatalog so it has to happen before
>> the
>> >> > branching.  Most likely at the beginning of next week.
>> >> >
>> >> > Thanks
>> >> >
>> >> >
>> >> > On Wed, Aug 28, 2013 at 5:24 PM, Eugene Koifman <
>> >> [EMAIL PROTECTED]
>> >> > >wrote:
>> >> >
>> >> > > Hi,
>> >> > >
>> >> > >
>> >> > > Here is the plan for refactoring HCatalog as was agreed to when it
>> was
>> >> > > merged into Hive during.  HIVE-4869 is the umbrella bug for this
>> work.
>> >> >  The
>> >> > > changes are complex and touch every single file under hcatalog.
>>  Please
>> >> > > comment.
>> >> > >
>> >> > > When HCatalog project was merged into Hive on 0.11 several
>> integration
>> >> > > items did not make the 0.11 deadline.  It was agreed to finish them
>> in
>> >> > 0.12
>> >> > > release.  Specifically:
>> >> > >
>> >> > > 1. HIVE-4895 - change package name from org.apache.hcatalog to
>> >> > > org.apache.hive.hcatalog
>> >> > >
>> >> > > 2. HIVE-4896 - create binary backwards compatibility layer for hcat
>> >> users
>> >> > > upgrading from 0.11 to 0.12
>> >> > >
>> >> > > For item 1, we’ll just move every file under org.apache.hcatalog to
>> >> > > org.apache.hive.hcatalog and update all “package” and “import”
>> >> statement
>> >> > as
>> >> > > well as all hcat/webhcat scripts.  This will include all JUnit
>> tests.
>> >> > >
>> >> > > Item 2 will ensure that if a user has a M/R program or Pig script,
>> etc.
>> >> > > that uses HCatalog public API, their programs will continue to work
>> w/o
>> >> > > change with hive 0.12.
>> >> > >
>> >> > > The proposal is to make the changes that have as little impact on
>> the
>> >> > > build system, in part to make upcoming ‘mavenization’ of hive
>> easier,
>> >> in
>> >> > > part to make the changes more manageable.
>> >> > >
>> >> > >
>> >> > >
>> >> > > The list of public interfaces (and their transitive closure) for
>> which
>> >> > > backwards compat will be provided.
>> >> > >
>> >> > >    1.
>> >> > >
>> >> > >    HCatLoader
>> >> > >    2.
>> >> > >
>> >> > >    HCatStorer
>> >> > >    3.
>> >> > >
>> >> > >    HCatInputFormat
>> >> > >    4.
>> >> > >
>> >> > >    HCatOutputFormat
>> >> > >    5.
>> >> > >
>> >> > >    HCatReader
>> >> > >    6.
>> >> > >
>> >> > >    HCatWriter
>> >> > >    7.
>> >> > >
>> >> > >    HCatRecord
>> >> > >    8.
>> >> > >
>> >> > >    HCatSchema
>> >> > >
>> >> > >
>> >> > > To achieve this, 0.11 version of these classes will be added in
>> >> > > org.apache.hcatalog package (after item 1 is done).  Each of these

Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org