Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> RFC: Major HCatalog refactoring


Copy link to this message
-
Re: RFC: Major HCatalog refactoring
OK that should be fine.  Though I would echo Edwards sentiment about
adding so much test time. Do these tests have to run each time? Does
it make sense to have an test target such as test-all-hcatalog and
then have then run them periodically manually, especially before
releases?

On Mon, Sep 2, 2013 at 10:36 AM, Eugene Koifman
<[EMAIL PROTECTED]> wrote:
> These will be new (I.e. 0.11 version) test classes which will be in the old
> org.apache.hcatalog package.  How does that affect the new framework?
>
> On Saturday, August 31, 2013, Brock Noland wrote:
>
>> Will these be new Java class files or new test methods to existing
>> classes?  I am just curious as to how this will play into the
>> distributed testing framework.
>>
>> On Sat, Aug 31, 2013 at 10:19 AM, Eugene Koifman
>> <[EMAIL PROTECTED]> wrote:
>> > not quite double but close  (on my Mac that means it will go up from 35
>> > minutes to 55-60) so in greater scheme of things it should be negligible
>> >
>> >
>> >
>> > On Sat, Aug 31, 2013 at 7:35 AM, Edward Capriolo <[EMAIL PROTECTED]
>> >wrote:
>> >
>> >> By coverage do you mean to say that:
>> >>
>> >> > Thus, the published HCatalog JARs will contain both packages and the
>> unit
>> >> > tests will cover both versions of the API.
>> >>
>> >> We are going to double the time of unit tests for this module?
>> >>
>> >>
>> >> On Fri, Aug 30, 2013 at 8:41 PM, Eugene Koifman <
>> [EMAIL PROTECTED]
>> >> >wrote:
>> >>
>> >> > This will change every file under hcatalog so it has to happen before
>> the
>> >> > branching.  Most likely at the beginning of next week.
>> >> >
>> >> > Thanks
>> >> >
>> >> >
>> >> > On Wed, Aug 28, 2013 at 5:24 PM, Eugene Koifman <
>> >> [EMAIL PROTECTED]
>> >> > >wrote:
>> >> >
>> >> > > Hi,
>> >> > >
>> >> > >
>> >> > > Here is the plan for refactoring HCatalog as was agreed to when it
>> was
>> >> > > merged into Hive during.  HIVE-4869 is the umbrella bug for this
>> work.
>> >> >  The
>> >> > > changes are complex and touch every single file under hcatalog.
>>  Please
>> >> > > comment.
>> >> > >
>> >> > > When HCatalog project was merged into Hive on 0.11 several
>> integration
>> >> > > items did not make the 0.11 deadline.  It was agreed to finish them
>> in
>> >> > 0.12
>> >> > > release.  Specifically:
>> >> > >
>> >> > > 1. HIVE-4895 - change package name from org.apache.hcatalog to
>> >> > > org.apache.hive.hcatalog
>> >> > >
>> >> > > 2. HIVE-4896 - create binary backwards compatibility layer for hcat
>> >> users
>> >> > > upgrading from 0.11 to 0.12
>> >> > >
>> >> > > For item 1, we’ll just move every file under org.apache.hcatalog to
>> >> > > org.apache.hive.hcatalog and update all “package” and “import”
>> >> statement
>> >> > as
>> >> > > well as all hcat/webhcat scripts.  This will include all JUnit
>> tests.
>> >> > >
>> >> > > Item 2 will ensure that if a user has a M/R program or Pig script,
>> etc.
>> >> > > that uses HCatalog public API, their programs will continue to work
>> w/o
>> >> > > change with hive 0.12.
>> >> > >
>> >> > > The proposal is to make the changes that have as little impact on
>> the
>> >> > > build system, in part to make upcoming ‘mavenization’ of hive
>> easier,
>> >> in
>> >> > > part to make the changes more manageable.
>> >> > >
>> >> > >
>> >> > >
>> >> > > The list of public interfaces (and their transitive closure) for
>> which
>> >> > > backwards compat will be provided.
>> >> > >
>> >> > >    1.
>> >> > >
>> >> > >    HCatLoader
>> >> > >    2.
>> >> > >
>> >> > >    HCatStorer
>> >> > >    3.
>> >> > >
>> >> > >    HCatInputFormat
>> >> > >    4.
>> >> > >
>> >> > >    HCatOutputFormat
>> >> > >    5.
>> >> > >
>> >> > >    HCatReader
>> >> > >    6.
>> >> > >
>> >> > >    HCatWriter
>> >> > >    7.
>> >> > >
>> >> > >    HCatRecord
>> >> > >    8.
>> >> > >
>> >> > >    HCatSchema
>> >> > >
>> >> > >
>> >> > > To achieve this, 0.11 version of these classes will be added in
>> >> > > org.apache.hcatalog package (after item 1 is done).  Each of these

Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB