Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Adding jar files when running hive in hwi mode or hiveserver mode


Copy link to this message
-
RE: Adding jar files when running hive in hwi mode or hiveserver mode
Ashish Thusoo 2009-08-26, 19:16
+1 for ajaxing this baby.

On the broader question of whether we should combine HWI and HiveServer - I think there are definite deployment and code reuse advantages in doing so, however keeping them separate also has the advantage that we can cluster HiveServers independently from HWI. Since the HiveServer sits in the data path, the independent scaling may have advantages. I am not sure how strong of an argument that is to not put them together. Simplicity obviously indicates that we should have them together.

Thoughts?

Ashish

-----Original Message-----
From: Edward Capriolo [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, August 26, 2009 9:45 AM
To: [EMAIL PROTECTED]
Subject: Re: Adding jar files when running hive in hwi mode or hiveserver mode

On Tue, Aug 25, 2009 at 8:13 PM, Vijay<[EMAIL PROTECTED]> wrote:
> Yep, I got it and now it works perfectly! I like hwi btw! It
> definitely makes things easier for a wider audience to try out hive.
> Your new session result bucket idea is very nice as well. I will keep
> trying more things and see if anything else comes up but so far it looks great!
> Thanks Edward!
>
> On Tue, Aug 25, 2009 at 7:25 AM, Edward Capriolo
> <[EMAIL PROTECTED]>
> wrote:
>>
>> On Tue, Aug 25, 2009 at 10:18 AM, Edward
>> Capriolo<[EMAIL PROTECTED]>
>> wrote:
>> > On Mon, Aug 24, 2009 at 10:13 PM, Vijay<[EMAIL PROTECTED]> wrote:
>> >> Probably spoke too soon :) I added this comment to the JIRA ticket
>> >> above.
>> >>
>> >> Hi, I tried the latest patch on trunk and there seems to be a problem.
>> >>
>> >> I was interested in using the "add jar " command to add jar files
>> >> to the path. However, by the time the command flows through the
>> >> SessionState to the AddResourceProcessor (in
>> >>
>> >> ./ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProc
>> >> essor.java), the command word "add" is not being stripped so the
>> >> resource processor is trying to find a ResourceType of "ADD."
>> >>
>> >> I'm not sure if this was an existing bug or was a result of the
>> >> current set of changes.
>> >>
>> >> [ Show > ]
>> >> Vijay added a comment - 24/Aug/09 07:12 PM Hi, I tried the latest
>> >> patch on trunk and there seems to be a problem. I was interested
>> >> in using the "add jar " command to add jar files to the path.
>> >> However, by the time the command flows through the SessionState to
>> >> the AddResourceProcessor (in
>> >>
>> >> ./ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProc
>> >> essor.java), the command word "add" is not being stripped so the
>> >> resource processor is trying to find a ResourceType of "ADD." I'm
>> >> not sure if this was an existing bug or was a result of the
>> >> current set of changes.
>> >> On Mon, Aug 24, 2009 at 5:30 PM, Vijay <[EMAIL PROTECTED]> wrote:
>> >>>
>> >>> That's awesome and looks like exactly what I needed. Local file
>> >>> system requirement is perfectly ok for now. I will check it out right away!
>> >>> Hopefully it will be checked in soon.
>> >>>
>> >>> Thanks Edward!
>> >>>
>> >>> On Mon, Aug 24, 2009 at 5:14 PM, Edward Capriolo
>> >>> <[EMAIL PROTECTED]>
>> >>> wrote:
>> >>>>
>> >>>> On Mon, Aug 24, 2009 at 8:09 PM, Prasad
>> >>>> Chakka<[EMAIL PROTECTED]>
>> >>>> wrote:
>> >>>> > Vijay, there is no solution for it yet. There may be a jira
>> >>>> > open but AFAIK, no one is working on it. You are welcome to
>> >>>> > contribute this feature.
>> >>>> >
>> >>>> > Prasad
>> >>>> >
>> >>>> >
>> >>>> > ________________________________
>> >>>> > From: Vijay <[EMAIL PROTECTED]>
>> >>>> > Reply-To: <[EMAIL PROTECTED]>
>> >>>> > Date: Mon, 24 Aug 2009 16:59:28 -0700
>> >>>> > To: <[EMAIL PROTECTED]>
>> >>>> > Subject: Re: Adding jar files when running hive in hwi mode or
>> >>>> > hiveserver mode
>> >>>> >
>> >>>> > Hi, is there any solution for this? How does everybody include
>> >>>> > custom jar files running hive in a non-cli mode?
>> >>>> >
>> >>>> > Thanks in advance,

Vijay,
That was always the goal. I often wonder which direction we should take HWI in.
Should HWI have some REST-ful stubs to turn it into a remote job submission system?
HiveServer uses thrift and I believe thrift has an HTTP-Transport so you might not need HWI to provide this.

Should we ajax things like the result bucket or the entire interface so it has that ooo aaahhh effect?

Really the larger question HWI has it's own multi-session management, HiveServer has this as well (now way back when it did not) . Should HWI just front end HiveServer?

Does anyone have any thoughts?
Edward