Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> [Discuss] project chop up


Copy link to this message
-
Re: [Discuss] project chop up
My mistake on saying hcat was a fork metastore. I had a brain fart for a
moment.

One way we could do this is create a folder called downstream. In our
release step we can execute the downstream builds and then copy the files
we need back. So nothing downstream will be on the classpath of the main
project.

This could help us breakup ql as well. Things like exotic file formats ,
and things that are pluggable like zk locking can go here. That might be
overkill.

For now we can focus on building downstream and hivethrift1might be the
first thing to try to downstream.
On Friday, July 26, 2013, Thejas Nair <[EMAIL PROTECTED]> wrote:
> +1 to the idea of making the build of core hive and other downstream
> components independent.
>
> bq.  I was under the impression that Hcat and hive-metastore was
> supposed to merge up somehow.
>
> The metastore code was never forked. Hcat was just using
> hive-metastore and making the metadata available to rest of hadoop
> (pig, java MR..).
> A lot of the changes that were driven by hcat goals were being made in
> hive-metastore. You can think of hcat as set of libraries that let pig
> and java MR use hive metastore. Since hcat is closely tied to
> hive-metastore, it makes sense to have them in same project.
>
>
> On Fri, Jul 26, 2013 at 6:33 AM, Edward Capriolo <[EMAIL PROTECTED]>
wrote:
>> Also i believe hcatalog web can fall into the same designation.
>>
>> Question , hcatalog was initily a big hive-metastore fork. I was under
the
>> impression that Hcat and hive-metastore was supposed to merge up somehow.
>> What is the status on that? I remember that was one of the core reasons
we
>> brought it in.
>>
>> On Friday, July 26, 2013, Edward Capriolo <[EMAIL PROTECTED]> wrote:
>>> I prefer option 3 as well.
>>>
>>>
>>> On Fri, Jul 26, 2013 at 12:52 AM, Brock Noland <[EMAIL PROTECTED]>
wrote:
>>>>
>>>> On Thu, Jul 25, 2013 at 9:48 PM, Edward Capriolo <[EMAIL PROTECTED]
>>>wrote:
>>>>
>>>> > I have been developing my laptop on a duel core 2 GB Ram laptop for
>> years
>>>> > now. With the addition of hcatalog, hive-thrift2, and some other
growth
>>>> > trying to develop hive in a eclipse on this machine craws, especially
>> if
>>>> > 'build automatically' is turned on. As we look to add on more things
>> this
>>>> > is only going to get worse.
>>>> >
>>>> > I am also noticing issues like this:
>>>> >
>>>> > https://issues.apache.org/jira/browse/HIVE-4849
>>>> >
>>>> > What I think we should do is strip down/out optional parts of hive.
>>>> >
>>>> > 1) Hive Hbase
>>>> >  This should really be it's own project to do this right we really
>> have to
>>>> > have multiple branches since hbase is not backwards compatible.
>>>> >
>>>> > 2) Hive Web Interface
>>>> > Now really a big project but not really critical can be just as
easily
>> be
>>>> > build separately
>>>> >
>>>> > 3) hive thrift 1
>>>> > We have hive thrift 2 now, it is time for the sun to set on
>> hivethrift1,
>>>> >
>>>> > 4) odbc
>>>> > Not entirely convinced about this one but it is really not critical
to
>>>> > running hive.
>>>> >
>>>> > What I think we should do is create sub-projects for the above things
>> or
>>>> > simply move them into directories that do not build with hive.
Ideally
>> they
>>>> > would use maven to pull dependencies.
>>>> >
>>>> > What does everyone think?
>>>> >
>>>>
>>>> I agree that projects like the HBase handler and probably others as
well
>>>> should somehow be "downstream" projects which simply depend on the hive
>>>> jars.  I see a couple alternatives for this:
>>>>
>>>> * Take the "module" in question to the Apache Incubator
>>>> * Move the "module" in question to the Apache Extras
>>>> * Breakup the projects within our own source tree
>>>>
>>>> I'd prefer the third option at this point.
>>>>
>>>> Brock
>>>>
>>>>
>>>>
>>>> Brock
>>>
>>>
>