Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> Review Request 14719: Better thread management for HiveServer2 async threads


Copy link to this message
-
Re: Review Request 14719: Better thread management for HiveServer2 async threads

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14719/#review27464
-----------------------------------------------------------

service/src/java/org/apache/hive/service/cli/session/SessionManager.java
<https://reviews.apache.org/r/14719/#comment53385>

    From the docs it seems that when maxThreads > coreThreads, the extra threads are killed after the keepalive time. Setting allowCoreThreadTimeOut to true applies the same policy to core threads (http://bit.ly/1igminx). So this acts like a fixedThreadPoolExecutor but with a keepalive time.
    
    If we use a bounded queue and different core & max sizes, when #requests > #core threads, new requests will get queued and only when the queue is full, it will create new threads till max threads is reached and then reject new tasks that are submitted. In case of long running queries, if #core threads is small, queuing might happen much earlier than expected and unless the queue size is small, new threads will not be created till the queue is full. This looked like a little complicated to tune and kind of not exactly fitting our requirements, since to tune it best, we might just end up tuning #core threads and queue size. What are your thoughts?
    
    But I agree on using a bounded queue and communicating the resource outage to the client rather than silent queueing - let me look at SynchronousQueue as well.  
    
    Thanks for the feedback!
- Vaibhav Gumashta
On Oct. 17, 2013, 8:32 p.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14719/
> -----------------------------------------------------------
>
> (Updated Oct. 17, 2013, 8:32 p.m.)
>
>
> Review request for hive and Thejas Nair.
>
>
> Bugs: HIVE-5229
>     https://issues.apache.org/jira/browse/HIVE-5229
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Improves the current strategy by not keeping all the async threads alive when there are no corresponding requests. The async threads terminate after a configurable keepalive time if there are no new requests to handle.
>
>
> Diffs
> -----
>
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fb3570b
>   conf/hive-default.xml.template b7234b4
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java f392d62
>
> Diff: https://reviews.apache.org/r/14719/diff/
>
>
> Testing
> -------
>
> TestEmbeddedThriftBinaryCLIService, TestThriftBinaryCLIService, TestJdbcDriver2
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>