Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Hive and thrift session help


Copy link to this message
-
Re: Hive and thrift session help
I get that HWI does manage sessions but it does that leveraging the internal
functionality of the "server." One usage pattern I'd like is some kind of a
"job" API. What I mean by that is an API that lets us simply submit a query,
get some kind of "job id," and leave. After that we use other APIs to query
the job status, kill it, get the output once it is done, etc. If we have a
simple API like this and the semantics to support this within hive, then the
UI can be completely decoupled and be as stateless as it can (using vanilla
apache+php as an example, we can't really do threads or stay resident after
submitting a job). Does something like this exist either within hive or at
the hadoop level? It seems to me may be this is something that needs to be
built first.

Thanks,
Vijay

On Tue, Sep 8, 2009 at 2:52 PM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> On Tue, Sep 8, 2009 at 5:15 PM, Royce
> Rollins<[EMAIL PROTECTED]> wrote:
> > OK I see. I just looked at the code in HWISessionManager.java.  So it
> looks
> > like either I will have to write my own ruby HWISessionManager that
> manages
> > sessions through thrift or expose the existng HWISessionManager via some
> web
> > service interface.  Has anyone done this?
> >
> > Royce
> >
> >
> > On 9/8/09 1:47 PM, "Edward Capriolo" <[EMAIL PROTECTED]> wrote:
> >
> >> On Tue, Sep 8, 2009 at 4:38 PM, Vijay<[EMAIL PROTECTED]> wrote:
> >>> Sorry to inject into this thread but I have the same problem (only I'm
> >>> trying to use the thrift PHP libraries from apache-php scripts). The
> problem
> >>> with this approach is that the http request cannot run indefinitely as
> the
> >>> server is executing a query. Are there any solutions for this?
> >>>
> >>> Thanks,
> >>> Vijay
> >>>
> >>> On Tue, Sep 8, 2009 at 1:35 PM, Royce Rollins <
> [EMAIL PROTECTED]>
> >>> wrote:
> >>>>
> >>>> Raghu,
> >>>> Thanks for the quick response.
> >>>> Yes.  My application is web based so instead of having to build some
> kind
> >>>> of
> >>>> session model myself for queries that might take a while,  I'd like to
> use
> >>>> a session model in the hive service.
> >>>>
> >>>> Royce
> >>>>
> >>>>
> >>>> On 9/8/09 1:32 PM, "Raghu Murthy" <[EMAIL PROTECTED]> wrote:
> >>>>
> >>>>> Our model so far has been to create a new connection to the hive
> thrift
> >>>>> server per session. Is there anything specific you are looking for in
> >>>>> sessions?
> >>>>>
> >>>>>
> >>>>> On 9/8/09 1:06 PM, "Royce Rollins" <[EMAIL PROTECTED]>
> wrote:
> >>>>>
> >>>>>> I¹m curently working on an application that connects to hive via the
> >>>>>> thrift
> >>>>>> ruby libraries.
> >>>>>>
> >>>>>> Does hive support creation of sessions using those libraries.  If
> so,
> >>>>>> how?
> >>>>>>
> >>>>>>
> >>>>>> Royce
> >>>>>
> >>>>
> >>>
> >>>
> >>
> >> Royce,
> >>
> >> The Hive Web Interface deals with this by having a threaded object
> >> (HWISessionManager) in the Web application scope. I am not sure if PHP
> >> has any equivalent to threading and Application Scope.
> >>
> >> Edward
> >
> >
>
> Someone correct me if I am wrong.
>
> Royce,
>
> You may be able to get at this another way. From my understanding, the
> internal hive web interface used at facebook would spawn ` bin/hive -e
> 'INSERT INTO X select * FROM`. All results were written to a hive
> table.
>
> Doing it this way gives you no way to interact with the query and
> 'stream' the result, set you can't really use 'fetchOne()' or
> 'fetchAll()' but you could start a query and set flags on completion.
>
> As for web interface, we just had some talks, and one of the things I
> was looking to do was create some type of web service style bindings.
> (We would also like to have HWI talk to Thrift and have thrift be the
> code path for everything). However, if we do make some web server
> style bindings they would really be independent of the back end. Do
> you want to work on this ? I would like to open a Jira and tackle the
> issue.
>
>
> The big picture here is that we need a 'state holder'. That is really
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB