Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # dev - [Discuss] project chop up


Copy link to this message
-
Re: [Discuss] project chop up
Xuefu Zhang 2013-08-17, 03:16
Thanks, Edward.

I'm big +1 to mavenize Hive. Hive has long reached a point where it's hard
to manage its build using ant. I'd like to help on this too.

Thanks,
Xuefu
On Fri, Aug 16, 2013 at 7:31 PM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> For those interested in pitching in.
> https://github.com/edwardcapriolo/hive
>
>
>
> On Fri, Aug 16, 2013 at 11:58 AM, Edward Capriolo <[EMAIL PROTECTED]
> >wrote:
>
> > Summary from hive-irc channel. Minor edits for spell check/grammar.
> >
> > The last 10 lines are a summary of the key points.
> >
> > [10:59:17] <ecapriolo> noland: et all. Do you want to talk about hive in
> > maven?
> > [11:01:06] smonchi [~
> > [EMAIL PROTECTED]] has quit IRC:
> > Quit: ... 'cause there is no patch for human stupidity ...
> > [11:10:04] <noland> ecapriolo: yeah that sounds good to me!
> > [11:10:22] <noland> I saw you created the jira but haven't had time to
> look
> > [11:10:32] <ecapriolo> So I found a few things
> > [11:10:49] <ecapriolo> In common there is one or two testats that
> actually
> > fork a process :)
> > [11:10:56] <ecapriolo> and use build.test.resources
> > [11:11:12] <ecapriolo> Some serde, uses some methods from ql in testing
> > [11:11:27] <ecapriolo> and shims really needs a separate hadoop test shim
> > [11:11:32] <ecapriolo> But that is all simple stuff
> > [11:11:47] <ecapriolo> The biggest problem is I do not know how to solve
> > shims with maven
> > [11:11:50] <ecapriolo> do you have any ideas
> > [11:11:52] <ecapriolo> ?
> > [11:13:00] <noland> That one is going to be a challenge. It might be that
> > in that section we have to drop down to ant
> > [11:14:44] <noland> Is it a requirement that we build both the .20 and
> .23
> > shims for a "package" as we do today?
> > [11:16:46] <ecapriolo> I was thinking we can do it like a JDBC driver
> > [11:16:59] <ecapriolo> Se separate out the interface of shims
> > [11:17:22] <ecapriolo> And then at runtime we drop in a driver
> implementing
> > [11:17:34] Wertax [~[EMAIL PROTECTED]] has quit IRC: Remote host
> > closed the connection
> > [11:17:36] <ecapriolo> That or we could use maven's profile system
> > [11:18:09] <ecapriolo> It seems that everything else can actually link
> > against hadoop-0.20.2 as a provided dependency
> > [11:18:37] <noland> Yeah either would work. The driver method would
> > probably require use to use ant build both the drivers?
> > [11:18:44] <noland> I am a fan of mvn profiles
> > [11:19:05] <ecapriolo> I was thinking we kinda separate the shim out into
> > its own project,, not a module
> > [11:19:10] <ecapriolo> to achive that jdbc thing
> > [11:19:27] <ecapriolo> But I do not have a solution yet, I was looking to
> > farm that out to someone smart...like you :)
> > [11:19:33] <noland> :)
> > [11:19:47] <ecapriolo> All I know is that we need a test shim because
> > HadoopShim requires hadoop-test jars
> > [11:20:10] <ecapriolo> then the Mini stuff is only used in qtest anyway
> > [11:20:48] <ecapriolo> Is this something you want to help with? I was
> > thinking of spinning up a github
> > [11:20:50] <noland> I think that the separate projects would work and
> > perhaps nicely.
> > [11:21:01] <noland> Yeah I'd be interested in helping!
> > [11:21:17] <noland> But I am going on vacation starting next week for
> > about 10 days
> > [11:21:27] <ecapriolo> Ah cool where are you going?
> > [11:21:37] <noland> Netherlands
> > [11:21:42] <noland> Biking around and such
> > [11:23:52] <noland> The one thing I was thinking about with regards to a
> > branch is keeping history. We'll want to keep history for the files but
> > AFAICT svn doesn't understand git mv.
> > [11:24:16] Wertax [~[EMAIL PROTECTED]] has joined #hive
> > [11:31:19] jeromatron [~[EMAIL PROTECTED]] has
> > quit IRC: Quit: My MacBook Pro has gone to sleep. ZZZzzz…
> > [11:35:49] <ecapriolo> noland: Right I do not play to suggest that we
> will
> > do this in git