Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> Review Request 14719: Better thread management for HiveServer2 async threads


Copy link to this message
-
Re: Review Request 14719: Better thread management for HiveServer2 async threads

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14719/#review27449
-----------------------------------------------------------

service/src/java/org/apache/hive/service/cli/session/SessionManager.java
<https://reviews.apache.org/r/14719/#comment53340>

    The timeout is applicable only if the number of threads are more than corePoolSize. But since corePoolSize==maxPoolSize, it won't have any impact.
    
    I think we can set the max to configurable value (we should change the config param name to hive.server2.async.exec.max.threads and use the old config name as a deprecated alias). Also add a config param for min, and use that as the value for corePoolSize.
    The default value of timeout I think can be in minutes (say 3 mins?), which is more representative of longer gaps that might be there between queries that come from a users session.  How about using a max default of 200 and a min of 5 .
    
    Also, I think we should also use a bounded queue. It is better to give an error that makes sense, rather than wait for running out of resources and giving a hard to decipher error. We should also handle rejections from the thread pool, when the limit has been hit.
- Thejas Nair
On Oct. 17, 2013, 8:32 p.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14719/
> -----------------------------------------------------------
>
> (Updated Oct. 17, 2013, 8:32 p.m.)
>
>
> Review request for hive and Thejas Nair.
>
>
> Bugs: HIVE-5229
>     https://issues.apache.org/jira/browse/HIVE-5229
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Improves the current strategy by not keeping all the async threads alive when there are no corresponding requests. The async threads terminate after a configurable keepalive time if there are no new requests to handle.
>
>
> Diffs
> -----
>
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fb3570b
>   conf/hive-default.xml.template b7234b4
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java f392d62
>
> Diff: https://reviews.apache.org/r/14719/diff/
>
>
> Testing
> -------
>
> TestEmbeddedThriftBinaryCLIService, TestThriftBinaryCLIService, TestJdbcDriver2
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>