Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> [Discuss] project chop up


Copy link to this message
-
Re: [Discuss] project chop up
For those interested in pitching in.
https://github.com/edwardcapriolo/hive

On Fri, Aug 16, 2013 at 11:58 AM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> Summary from hive-irc channel. Minor edits for spell check/grammar.
>
> The last 10 lines are a summary of the key points.
>
> [10:59:17] <ecapriolo> noland: et all. Do you want to talk about hive in
> maven?
> [11:01:06] smonchi [~
> [EMAIL PROTECTED]] has quit IRC:
> Quit: ... 'cause there is no patch for human stupidity ...
> [11:10:04] <noland> ecapriolo: yeah that sounds good to me!
> [11:10:22] <noland> I saw you created the jira but haven't had time to look
> [11:10:32] <ecapriolo> So I found a few things
> [11:10:49] <ecapriolo> In common there is one or two testats that actually
> fork a process :)
> [11:10:56] <ecapriolo> and use build.test.resources
> [11:11:12] <ecapriolo> Some serde, uses some methods from ql in testing
> [11:11:27] <ecapriolo> and shims really needs a separate hadoop test shim
> [11:11:32] <ecapriolo> But that is all simple stuff
> [11:11:47] <ecapriolo> The biggest problem is I do not know how to solve
> shims with maven
> [11:11:50] <ecapriolo> do you have any ideas
> [11:11:52] <ecapriolo> ?
> [11:13:00] <noland> That one is going to be a challenge. It might be that
> in that section we have to drop down to ant
> [11:14:44] <noland> Is it a requirement that we build both the .20 and .23
> shims for a "package" as we do today?
> [11:16:46] <ecapriolo> I was thinking we can do it like a JDBC driver
> [11:16:59] <ecapriolo> Se separate out the interface of shims
> [11:17:22] <ecapriolo> And then at runtime we drop in a driver implementing
> [11:17:34] Wertax [~[EMAIL PROTECTED]] has quit IRC: Remote host
> closed the connection
> [11:17:36] <ecapriolo> That or we could use maven's profile system
> [11:18:09] <ecapriolo> It seems that everything else can actually link
> against hadoop-0.20.2 as a provided dependency
> [11:18:37] <noland> Yeah either would work. The driver method would
> probably require use to use ant build both the drivers?
> [11:18:44] <noland> I am a fan of mvn profiles
> [11:19:05] <ecapriolo> I was thinking we kinda separate the shim out into
> its own project,, not a module
> [11:19:10] <ecapriolo> to achive that jdbc thing
> [11:19:27] <ecapriolo> But I do not have a solution yet, I was looking to
> farm that out to someone smart...like you :)
> [11:19:33] <noland> :)
> [11:19:47] <ecapriolo> All I know is that we need a test shim because
> HadoopShim requires hadoop-test jars
> [11:20:10] <ecapriolo> then the Mini stuff is only used in qtest anyway
> [11:20:48] <ecapriolo> Is this something you want to help with? I was
> thinking of spinning up a github
> [11:20:50] <noland> I think that the separate projects would work and
> perhaps nicely.
> [11:21:01] <noland> Yeah I'd be interested in helping!
> [11:21:17] <noland> But I am going on vacation starting next week for
> about 10 days
> [11:21:27] <ecapriolo> Ah cool where are you going?
> [11:21:37] <noland> Netherlands
> [11:21:42] <noland> Biking around and such
> [11:23:52] <noland> The one thing I was thinking about with regards to a
> branch is keeping history. We'll want to keep history for the files but
> AFAICT svn doesn't understand git mv.
> [11:24:16] Wertax [~[EMAIL PROTECTED]] has joined #hive
> [11:31:19] jeromatron [~[EMAIL PROTECTED]] has
> quit IRC: Quit: My MacBook Pro has gone to sleep. ZZZzzz…
> [11:35:49] <ecapriolo> noland: Right I do not play to suggest that we will
> do this in git
> [11:36:11] <ecapriolo> I just see that we are going to have to hack stuff
> up and it is not the type of work that lends itself well to branches.
> [11:36:17] <noland> Ahh ok
> [11:36:56] <ecapriolo> Once we come up with a solution for the shims, and
> we have something that can reasonably build and test hive we can figure out
> how to apply that to a branch/trunk
> [11:36:58] <noland> yeah so just do a POC on github and then implement on
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB