Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Having a Connections Leak with the Hive Server


Copy link to this message
-
RE: Having a Connections Leak with the Hive Server
Bennie Schut 2010-09-02, 13:29
We had some leaking file descriptors which ended up being a problem in hadoop. They fixed it on 0.21 but not on older versions. There is a workaround for hive which we successfully use. By adding this to your hive-site.xml:

  <!-- workaround for connection leak problem fixed in HADOOP-5476 but only commited to hadoop 0.21.0 -->
  <property>
    <name>hive.fileformat.check</name>
    <value>false</value>
  </property>

Bennie.

________________________________
From: Dave Brondsema [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 02, 2010 3:13 PM
To: [EMAIL PROTECTED]
Subject: Re: Having a Connections Leak with the Hive Server

Scott, after re-reading your original email, I'm thinking maybe we didn't have the same problem.  Hive crashed for us when it ran out of file descriptors, it didn't hang.  Nonetheless, an upgrade may help.
On Wed, Sep 1, 2010 at 10:24 AM, Scott Whitecross <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Thanks Dave.   We've been investigating moving to .6, and this may help make the decision easier.
On Tue, Aug 31, 2010 at 5:02 PM, Dave Brondsema <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
We had that problem on 0.4 also.  0.6 seems to be working better now, but we only switched a day or two ago.  See https://issues.apache.org/jira/browse/HIVE-1181 and http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201001.mbox/%[EMAIL PROTECTED]%3E

On Mon, Aug 30, 2010 at 3:50 PM, Scott Whitecross <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi all -

I'm running the Hive Server to allow multiple JDBC queries and inserts over the course of a day.  What I've noticed is that there seems to be a pretty bad leak with connections over the course of a week, to the point of making the machine unreachable.  Looking through the code being used, it appears to be closing Hive connections appropriately.  Is there a known problem with Hive and connections now?  (I'm currently running Hive .4).

Thanks.
--
Dave Brondsema
Software Engineer
Geeknet

www.geek.net<http://www.geek.net>
--
Dave Brondsema
Software Engineer
Geeknet

www.geek.net<http://www.geek.net>