Hive, mail # user - Re: Hive Server Leaking File Descriptors?

Hive Server Leaking File Descriptors?

Andy Kent 2010-01-25, 11:13

We use the hive thrift server and ruby client to submit queries to hive. But we have noticed that the number of open file descriptors climbs steadily on the machine running hive.

On a cluster of 21 nodes with hive running for around 5 days and processing around 200 queries per day we see around 3,000 open file descriptors. This growth continues until it saturates the system limits and then stops responding to requests.

All of the connections appear to be to port 50010 (HDFS?) on the slave machines.

Any help or suggestions would be greatly appreciated.
Thanks, Andy.

Below is an sample netstat output...

deploy@hadoopmaster:~$ ps aux|grep HiveServer|grep -v grep\ HiveServer|awk '{print $2}'|while read i; do sudo netstat -np|grep \ "$i"\/java;done
tcp6 33380 0 192.168.31.200:41491 192.168.31.217:50010 ESTABLISHED 1768/java
tcp6 66800 0 192.168.31.200:39865 192.168.31.209:50010 ESTABLISHED 1768/java
tcp6 66800 0 192.168.31.200:33770 192.168.31.205:50010 ESTABLISHED 1768/java
tcp6 67040 0 192.168.31.200:45931 192.168.31.215:50010 ESTABLISHED 1768/java
tcp6 66880 0 192.168.31.200:55003 192.168.31.220:50010 ESTABLISHED 1768/java
tcp6 66760 0 192.168.31.200:52236 192.168.31.209:50010 ESTABLISHED 1768/java
tcp6 23033 0 192.168.31.200:51442 192.168.31.210:50010 ESTABLISHED 1768/java
tcp6 66504 0 192.168.31.200:34695 192.168.31.210:50010 ESTABLISHED 1768/java
tcp6 67040 0 192.168.31.200:45760 192.168.31.215:50010 ESTABLISHED 1768/java
tcp6 66680 0 192.168.31.200:55634 192.168.31.213:50010 ESTABLISHED 1768/java
tcp6 18536 0 192.168.31.200:47812 192.168.31.208:50010 ESTABLISHED 1768/java
tcp6 18536 0 192.168.31.200:55430 192.168.31.204:50010 ESTABLISHED 1768/java
tcp6 66680 0 192.168.31.200:33447 192.168.31.205:50010 ESTABLISHED 1768/java
tcp6 66680 0 192.168.31.200:44618 192.168.31.203:50010 ESTABLISHED 1768/java
tcp6 28269 0 192.168.31.200:60042 192.168.31.220:50010 ESTABLISHED 1768/java
tcp6 66560 0 192.168.31.200:60043 192.168.31.203:50010 ESTABLISHED 1768/java
tcp6 66760 0 192.168.31.200:38104 192.168.31.219:50010 ESTABLISHED 1768/java
tcp6 66760 0 192.168.31.200:36461 192.168.31.205:50010 ESTABLISHED 1768/java
tcp6 45333 0 192.168.31.200:43051 192.168.31.207:50010 ESTABLISHED 1768/java
tcp6 66680 0 192.168.31.200:32782 192.168.31.213:50010 ESTABLISHED 1768/java
tcp6 8794 0 192.168.31.200:33000 192.168.31.209:50010 ESTABLISHED 1768/java
tcp6 0 0 192.168.31.200:51397 192.168.31.200:50010 ESTABLISHED 1768/java
tcp6 66760 0 192.168.31.200:47584 192.168.31.206:50010 ESTABLISHED 1768/java
tcp6 66680 0 192.168.31.200:49115 192.168.31.219:50010 ESTABLISHED 1768/java
tcp6 66760 0 192.168.31.200:46942 192.168.31.207:50010 ESTABLISHED 1768/java
tcp6 66800 0 192.168.31.200:51521 192.168.31.212:50010 ESTABLISHED 1768/java
tcp6 66680 0 192.168.31.200:35897 192.168.31.205:50010 ESTABLISHED 1768/java
tcp6 42426 0 192.168.31.200:44042 192.168.31.208:50010 ESTABLISHED 1768/java
tcp6 37196 0 192.168.31.200:45567 192.168.31.212:50010 ESTABLISHED 1768/java
tcp6 66800 0 192.168.31.200:38376 192.168.31.207:50010 ESTABLISHED 1768/java
tcp6 66760 0 192.168.31.200:47279 192.168.31.203:50010 ESTABLISHED 1768/java
tcp6 66072 0 192.168.31.200:35648 192.168.31.217:50010 ESTABLISHED 1768/java
tcp6 66384 0 192.168.31.200:39264 192.168.31.206:50010 ESTABLISHED 1768/java
tcp6 8152 0 192.168.31.200:45371 192.168.31.209:50010 ESTABLISHED 1768/java
tcp6 66760 0 192.168.31.200:37021 192.168.31.209:50010 ESTABLISHED 1768/java
tcp6 34844 0 192.168.31.200:47316 192.168.31.219:50010 ESTABLISHED 1768/java
tcp6 66680 0 192.168.31.200:45311 192.168.31.215:50010 ESTABLISHED 1768/java
tcp6 66680 0 192.168.31.200:47921 192.168.31.219:50010 ESTABLISHED 1768/java
tcp6 66880 0 192.168.31.200:49766 192.168.31.220:50010 ESTABLISHED 1768/java
tcp6 47556 0 192.168.31.200:40457 192.168.31.208:50010 ESTABLISHED 1768/java
tcp6 67080 0 192.168.31.200:58490 192.168.31.220:50010 ESTABLISHED 1768/java
tcp6 66600 0 192.168.31.200:60526 192.168.31.203:50010 ESTABLISHED 1768/java
tcp6 66560 0 192.168.31.200:42714 192.168.31.220:50010 ESTABLISHED 1768/java
tcp6 66760 0 192.168.31.200:51175 192.168.31.201:50010 ESTABLISHED 1768/java
tcp6 66760 0 192.168.31.200:39403 192.168.31.206:50010 ESTABLISHED 1768/java
tcp6 66800 0 192.168.31.200:50538 192.168.31.206:50010 ESTABLISHED 1768/java
tcp6 66760 0 192.168.31.200:58341 192.168.31.219:50010 ESTABLISHED 1768/java
tcp6 66840 0 192.168.31.200:34861 192.168.31.204:50010 ESTABLISHED 1768/java
tcp6 18445 0 192.168.31.200:39932 192.168.31.202:50010 ESTABLISHED 1768/java
tcp6 20500 0 192.168.31.200:58142 192.168.31.206:50010 ESTABLISHED 1768/java
tcp6 66720 0 192.168.31.200:35214 192.168.31.218:50010 ESTABLISHED 1768/java
tcp6 66800 0 192.168.31.200:47101 192.168.31.215:50010 ESTABLISHED 1768/java
tcp6 15897 0 192.168.31.200:34267 192.168.31.202:50010 ESTABLISHED 1768/java
tcp6 66760 0 192.168.31.200:36748 192.168.31.218:50010 ESTABLISHED 1768/java
t

Re: Hive Server Leaking File Descriptors?

Jay Booth 2010-01-25, 13:59

That's the datanode port.. if I had to guess, Hive's connecting to DFS
directly for some reason (maybe for "select *" queries?) and not finishing
their reads or closing the connections after.

On Mon, Jan 25, 2010 at 6:13 AM, Andy Kent <[EMAIL PROTECTED]> wrote:

> We use the hive thrift server and ruby client to submit queries to hive.
> But we have noticed that the number of open file descriptors climbs
> steadily on the machine running hive.
>
> On a cluster of 21 nodes with hive running for around 5 days and processing
> around 200 queries per day we see around 3,000 open file descriptors. This
> growth continues until it saturates the system limits and then stops
> responding to requests.
>
> All of the connections appear to be to port 50010 (HDFS?) on the slave
> machines.
>
> Any help or suggestions would be greatly appreciated.
> Thanks, Andy.
>
> Below is an sample netstat output...
>
> deploy@hadoopmaster:~$ ps aux|grep HiveServer|grep -v grep\ HiveServer|awk
> '{print $2}'|while read i; do sudo netstat -np|grep \ "$i"\/java;done
> tcp6 33380 0 192.168.31.200:41491 192.168.31.217:50010
> ESTABLISHED 1768/java
> tcp6 66800 0 192.168.31.200:39865 192.168.31.209:50010
> ESTABLISHED 1768/java
> tcp6 66800 0 192.168.31.200:33770 192.168.31.205:50010
> ESTABLISHED 1768/java
> tcp6 67040 0 192.168.31.200:45931 192.168.31.215:50010
> ESTABLISHED 1768/java
> tcp6 66880 0 192.168.31.200:55003 192.168.31.220:50010
> ESTABLISHED 1768/java
> tcp6 66760 0 192.168.31.200:52236 192.168.31.209:50010
> ESTABLISHED 1768/java
> tcp6 23033 0 192.168.31.200:51442 192.168.31.210:50010
> ESTABLISHED 1768/java
> tcp6 66504 0 192.168.31.200:34695 192.168.31.210:50010
> ESTABLISHED 1768/java
> tcp6 67040 0 192.168.31.200:45760 192.168.31.215:50010
> ESTABLISHED 1768/java
> tcp6 66680 0 192.168.31.200:55634 192.168.31.213:50010
> ESTABLISHED 1768/java
> tcp6 18536 0 192.168.31.200:47812 192.168.31.208:50010
> ESTABLISHED 1768/java
> tcp6 18536 0 192.168.31.200:55430 192.168.31.204:50010
> ESTABLISHED 1768/java
> tcp6 66680 0 192.168.31.200:33447 192.168.31.205:50010
> ESTABLISHED 1768/java
> tcp6 66680 0 192.168.31.200:44618 192.168.31.203:50010
> ESTABLISHED 1768/java
> tcp6 28269 0 192.168.31.200:60042 192.168.31.220:50010
> ESTABLISHED 1768/java
> tcp6 66560 0 192.168.31.200:60043 192.168.31.203:50010
> ESTABLISHED 1768/java
> tcp6 66760 0 192.168.31.200:38104 192.168.31.219:50010
> ESTABLISHED 1768/java
> tcp6 66760 0 192.168.31.200:36461 192.168.31.205:50010
> ESTABLISHED 1768/java
> tcp6 45333 0 192.168.31.200:43051 192.168.31.207:50010
> ESTABLISHED 1768/java
> tcp6 66680 0 192.168.31.200:32782 192.168.31.213:50010
> ESTABLISHED 1768/java
> tcp6 8794 0 192.168.31.200:33000 192.168.31.209:50010
> ESTABLISHED 1768/java
> tcp6 0 0 192.168.31.200:51397 192.168.31.200:50010
> ESTABLISHED 1768/java
> tcp6 66760 0 192.168.31.200:47584 192.168.31.206:50010
> ESTABLISHED 1768/java
> tcp6 66680 0 192.168.31.200:49115 192.168.31.219:50010
> ESTABLISHED 1768/java
> tcp6 66760 0 192.168.31.200:46942 192.168.31.207:50010
> ESTABLISHED 1768/java
> tcp6 66800 0 192.168.31.200:51521 192.168.31.212:50010
> ESTABLISHED 1768/java
> tcp6 66680 0 192.168.31.200:35897 192.168.31.205:50010
> ESTABLISHED 1768/java
> tcp6 42426 0 192.168.31.200:44042 192.168.31.208:50010
> ESTABLISHED 1768/java
> tcp6 37196 0 192.168.31.200:45567 192.168.31.212:50010
> ESTABLISHED 1768/java
> tcp6 66800 0 192.168.31.200:38376 192.168.31.207:50010
> ESTABLISHED 1768/java
> tcp6 66760 0 192.168.31.200:47279 192.168.31.203:50010
> ESTABLISHED 1768/java
> tcp6 66072 0 192.168.31.200:35648 192.168.31.217:50010
> ESTABLISHED 1768/java

Re: Hive Server Leaking File Descriptors?

Andy Kent 2010-01-25, 14:06

On 25 Jan 2010, at 13:59, Jay Booth wrote:

> That's the datanode port.. if I had to guess, Hive's connecting to DFS directly for some reason (maybe for "select *" queries?) and not finishing their reads or closing the connections after.
Thanks for the response.

That's what I was suspecting. I have triple checked and our Ruby code and it is defiantly closing it's thrift connections properly.

I'll try running some different queries and see if I can suss out some examples of which ones are leaky. Is this something that I should post to Jira or is it a known issue? I can't believe other people haven't noticed this?

Re: Hive Server Leaking File Descriptors?

Jay Booth 2010-01-25, 15:11

Yeah, I'd guess that this is a Hive issue, although it could be a
combination.. maybe if you're doing queries and then closing your thrift
connection before reading all results, Hive doesn't know what to do and
leaves the connection open? Once the west coast folks wake up, they might
have a better answer for you than I do.

On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent <[EMAIL PROTECTED]> wrote:

> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>
> > That's the datanode port.. if I had to guess, Hive's connecting to DFS
> directly for some reason (maybe for "select *" queries?) and not finishing
> their reads or closing the connections after.
>
>
> Thanks for the response.
>
> That's what I was suspecting. I have triple checked and our Ruby code and
> it is defiantly closing it's thrift connections properly.
>
> I'll try running some different queries and see if I can suss out some
> examples of which ones are leaky. Is this something that I should post to
> Jira or is it a known issue? I can't believe other people haven't noticed
> this?

Re: Hive Server Leaking File Descriptors?

Jay Booth 2010-01-25, 15:30

Actually, we had an issue with this, it was a bug in SequenceFile where if
there were problems opening a file, it would leave a filehandle open and
never close it.

Here's the patch -- It's already fixed in 0.21/trunk, if I get some time
this week I'll submit it against 0.20.2 -- could you apply this to hadoop
and let me know if it fixes things for you?

On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth <[EMAIL PROTECTED]> wrote:

> Yeah, I'd guess that this is a Hive issue, although it could be a
> combination.. maybe if you're doing queries and then closing your thrift
> connection before reading all results, Hive doesn't know what to do and
> leaves the connection open? Once the west coast folks wake up, they might
> have a better answer for you than I do.
>
>
> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent <[EMAIL PROTECTED]>wrote:
>
>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>
>> > That's the datanode port.. if I had to guess, Hive's connecting to DFS
>> directly for some reason (maybe for "select *" queries?) and not finishing
>> their reads or closing the connections after.
>>
>>
>> Thanks for the response.
>>
>> That's what I was suspecting. I have triple checked and our Ruby code and
>> it is defiantly closing it's thrift connections properly.
>>
>> I'll try running some different queries and see if I can suss out some
>> examples of which ones are leaky. Is this something that I should post to
>> Jira or is it a known issue? I can't believe other people haven't noticed
>> this?
>
>
>

Re: Hive Server Leaking File Descriptors?

Andy Kent 2010-01-25, 15:36

I can give try and give it a go. I'm not convinced though as we are working with CSV files and don't touch sequence files at all at the moment.

We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and Hive 0.40
On 25 Jan 2010, at 15:30, Jay Booth wrote:

> Actually, we had an issue with this, it was a bug in SequenceFile where if there were problems opening a file, it would leave a filehandle open and never close it.
>
> Here's the patch -- It's already fixed in 0.21/trunk, if I get some time this week I'll submit it against 0.20.2 -- could you apply this to hadoop and let me know if it fixes things for you?
>
> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> Yeah, I'd guess that this is a Hive issue, although it could be a combination.. maybe if you're doing queries and then closing your thrift connection before reading all results, Hive doesn't know what to do and leaves the connection open? Once the west coast folks wake up, they might have a better answer for you than I do.
>
>
> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>
>> That's the datanode port.. if I had to guess, Hive's connecting to DFS directly for some reason (maybe for "select *" queries?) and not finishing their reads or closing the connections after.
>
>
> Thanks for the response.
>
> That's what I was suspecting. I have triple checked and our Ruby code and it is defiantly closing it's thrift connections properly.
>
> I'll try running some different queries and see if I can suss out some examples of which ones are leaky. Is this something that I should post to Jira or is it a known issue? I can't believe other people haven't noticed this?
>
>
> <SequenceFile.patch>

Re: Hive Server Leaking File Descriptors?

Bennie Schut 2010-02-15, 14:09

Did this help? I'm running into a similar problem. slowly leaking
connections to 50010 and after a hive restart all is ok again.

Andy Kent wrote:
> I can give try and give it a go. I'm not convinced though as we are working with CSV files and don't touch sequence files at all at the moment.
>
> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and Hive 0.40
>
>
> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>
>
>> Actually, we had an issue with this, it was a bug in SequenceFile where if there were problems opening a file, it would leave a filehandle open and never close it.
>>
>> Here's the patch -- It's already fixed in 0.21/trunk, if I get some time this week I'll submit it against 0.20.2 -- could you apply this to hadoop and let me know if it fixes things for you?
>>
>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> Yeah, I'd guess that this is a Hive issue, although it could be a combination.. maybe if you're doing queries and then closing your thrift connection before reading all results, Hive doesn't know what to do and leaves the connection open? Once the west coast folks wake up, they might have a better answer for you than I do.
>>
>>
>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>
>>
>>> That's the datanode port.. if I had to guess, Hive's connecting to DFS directly for some reason (maybe for "select *" queries?) and not finishing their reads or closing the connections after.
>>>
>> Thanks for the response.
>>
>> That's what I was suspecting. I have triple checked and our Ruby code and it is defiantly closing it's thrift connections properly.
>>
>> I'll try running some different queries and see if I can suss out some examples of which ones are leaky. Is this something that I should post to Jira or is it a known issue? I can't believe other people haven't noticed this?
>>
>>
>> <SequenceFile.patch>
>>
>
>

Re: Hive Server Leaking File Descriptors?

Andy Kent 2010-02-15, 15:42

Nope, no luck so far.

We have upped the number of file descriptors and are having to restart hive every week or so :(

Any other suggestions would be greatly appreciated.

On 15 Feb 2010, at 14:09, Bennie Schut wrote:

> Did this help? I'm running into a similar problem. slowly leaking
> connections to 50010 and after a hive restart all is ok again.
>
> Andy Kent wrote:
>> I can give try and give it a go. I'm not convinced though as we are working with CSV files and don't touch sequence files at all at the moment.
>>
>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and Hive 0.40
>>
>>
>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>
>>
>>> Actually, we had an issue with this, it was a bug in SequenceFile where if there were problems opening a file, it would leave a filehandle open and never close it.
>>>
>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get some time this week I'll submit it against 0.20.2 -- could you apply this to hadoop and let me know if it fixes things for you?
>>>
>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>>> Yeah, I'd guess that this is a Hive issue, although it could be a combination.. maybe if you're doing queries and then closing your thrift connection before reading all results, Hive doesn't know what to do and leaves the connection open? Once the west coast folks wake up, they might have a better answer for you than I do.
>>>
>>>
>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>
>>>
>>>> That's the datanode port.. if I had to guess, Hive's connecting to DFS directly for some reason (maybe for "select *" queries?) and not finishing their reads or closing the connections after.
>>>>
>>> Thanks for the response.
>>>
>>> That's what I was suspecting. I have triple checked and our Ruby code and it is defiantly closing it's thrift connections properly.
>>>
>>> I'll try running some different queries and see if I can suss out some examples of which ones are leaky. Is this something that I should post to Jira or is it a known issue? I can't believe other people haven't noticed this?
>>>
>>>
>>> <SequenceFile.patch>
>>>
>>
>>
>
>

Re: Hive Server Leaking File Descriptors?

Zheng Shao 2010-02-15, 22:42

Can you go to that box, sudo as root, and do "lsof | grep 12345" where
12345 is the process id of the hive server?
We should be able to see the names of the files that are open.

Zheng

On Mon, Feb 15, 2010 at 7:42 AM, Andy Kent <[EMAIL PROTECTED]> wrote:
> Nope, no luck so far.
>
> We have upped the number of file descriptors and are having to restart hive every week or so :(
>
> Any other suggestions would be greatly appreciated.
>
> On 15 Feb 2010, at 14:09, Bennie Schut wrote:
>
>> Did this help? I'm running into a similar problem. slowly leaking
>> connections to 50010 and after a hive restart all is ok again.
>>
>> Andy Kent wrote:
>>> I can give try and give it a go. I'm not convinced though as we are working with CSV files and don't touch sequence files at all at the moment.
>>>
>>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and Hive 0.40
>>>
>>>
>>> On 25 Jan 2010, at 15:30, Jay Booth wrote:
>>>
>>>
>>>> Actually, we had an issue with this, it was a bug in SequenceFile where if there were problems opening a file, it would leave a filehandle open and never close it.
>>>>
>>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get some time this week I'll submit it against 0.20.2 -- could you apply this to hadoop and let me know if it fixes things for you?
>>>>
>>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>>>> Yeah, I'd guess that this is a Hive issue, although it could be a combination.. maybe if you're doing queries and then closing your thrift connection before reading all results, Hive doesn't know what to do and leaves the connection open? Once the west coast folks wake up, they might have a better answer for you than I do.
>>>>
>>>>
>>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>>>> On 25 Jan 2010, at 13:59, Jay Booth wrote:
>>>>
>>>>
>>>>> That's the datanode port.. if I had to guess, Hive's connecting to DFS directly for some reason (maybe for "select *" queries?) and not finishing their reads or closing the connections after.
>>>>>
>>>> Thanks for the response.
>>>>
>>>> That's what I was suspecting. I have triple checked and our Ruby code and it is defiantly closing it's thrift connections properly.
>>>>
>>>> I'll try running some different queries and see if I can suss out some examples of which ones are leaky. Is this something that I should post to Jira or is it a known issue? I can't believe other people haven't noticed this?
>>>>
>>>>
>>>> <SequenceFile.patch>
>>>>
>>>
>>>
>>
>>
>
>

--
Yours,
Zheng

Re: Hive Server Leaking File Descriptors?

Andy Kent 2010-02-15, 23:24

I have included the output from lsof, Unfortunately though I only restarted the server a few mins ago and so it doesn't really illustrate the problem. I'll try to post back with the same command tomorrow so that we can diff between the two.

Thanks, Andy.

java 16908 hadoop cwd DIR 0,17 240 7037865 /var/run/hadoop-0.20
java 16908 hadoop rtd DIR 8,1 4096 2 /
java 16908 hadoop txt REG 8,1 50810 3555462 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java
java 16908 hadoop mem REG 8,1 127296 3443156 /usr/lib/liblzo2.so.2.0.0
java 16908 hadoop mem REG 8,1 69472 3735429 /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0
java 16908 hadoop mem REG 8,1 44257 3555570 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnio.so
java 16908 hadoop mem REG 8,1 80760 1164352 /lib/libresolv-2.7.so
java 16908 hadoop mem REG 8,1 22856 1164345 /lib/libnss_dns-2.7.so
java 16908 hadoop mem REG 8,1 1145980 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar
java 16908 hadoop mem REG 8,1 38105 3555614 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libmanagement.so
java 16908 hadoop mem REG 8,1 111901 3555569 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnet.so
java 16908 hadoop mem REG 8,1 392124 3532065 /usr/lib/hive/lib/velocity-1.5.jar
java 16908 hadoop mem REG 8,1 229928 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar
java 16908 hadoop mem REG 8,1 724225 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar
java 16908 hadoop mem REG 8,1 391834 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar
java 16908 hadoop mem REG 8,1 169666 3532090 /usr/lib/hive/lib/libthrift.jar
java 16908 hadoop mem REG 8,1 114342 3532078 /usr/lib/hive/lib/libfb303.jar
java 16908 hadoop mem REG 8,1 121070 3532069 /usr/lib/hive/lib/junit-3.8.1.jar
java 16908 hadoop mem REG 8,1 45946 3532058 /usr/lib/hive/lib/json.jar
java 16908 hadoop mem REG 8,1 87325 3532030 /usr/lib/hive/lib/jline-0.9.94.jar
java 16908 hadoop mem REG 8,1 179508 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar
java 16908 hadoop mem REG 8,1 20202 3532073 /usr/lib/hive/lib/hive_shims.jar
java 16908 hadoop mem REG 8,1 95488 3532068 /usr/lib/hive/lib/hive_service.jar
java 16908 hadoop mem REG 8,1 390114 3532076 /usr/lib/hive/lib/hive_serde.jar
java 16908 hadoop mem REG 8,1 500917 3532063 /usr/lib/hive/lib/hive_metastore.jar
java 16908 hadoop mem REG 8,1 31931 3532070 /usr/lib/hive/lib/hive_jdbc.jar
java 16908 hadoop mem REG 8,1 19349 3532081 /usr/lib/hive/lib/hive_hwi.jar
java 16908 hadoop mem REG 8,1 1798010 3532071 /usr/lib/hive/lib/hive_exec.jar
java 16908 hadoop mem REG 8,1 11412 3532091 /usr/lib/hive/lib/hive_contrib.jar
java 16908 hadoop mem REG 8,1 14099 3532074 /usr/lib/hive/lib/hive_common.jar
java 16908 hadoop mem REG 8,1 12641 3532064 /usr/lib/hive/lib/hive_cli.jar
java 16908 hadoop mem REG 8,1 7092 3532067 /usr/lib/hive/lib/hive_anttasks.jar
java 16908 hadoop mem REG 8,1 2446767 3532072 /usr/lib/hive/lib/derby.jar
java 16908 hadoop mem REG 8,1 1154170 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar
java 16908 hadoop mem REG 8,1 176174 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar
java 16908 hadoop mem REG 8,1 1923209 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar
java 16908 hadoop mem REG 8,1 26202 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
java 16908 hadoop mem REG 8,1 38015 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar
java 16908 hadoop mem REG 8,1 261809 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar
java 16908 hadoop mem REG 8,1 575389 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar
java 16908 hadoop mem REG 8,1 46725 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar
java 16908 hadoop mem REG 8,1 258337 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar
java 16908 hadoop mem REG 8,1 43035 3532057 /usr/lib/hive/lib/asm-3.1.jar
java 16908 hadoop mem REG 8,1 105209 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar
java 16908 hadoop mem REG 8,1 134910 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar
java 16908 hadoop mem REG 8,1 1024681 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar
java 16908 hadoop mem REG 8,1 15010 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar
java 16908 hadoop mem REG 8,1 8601 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar
java 16908 hadoop m

Re: Hive Server Leaking File Descriptors?

Bennie Schut 2010-02-16, 08:05

I think it's lines like this:
java 16908 hadoop 160u IPv6 153354839 TCP
hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
(ESTABLISHED)

We have the same.
Currently no queries running and many of those connections open. After
running some queries a few more are added to the list.

java 5888 dwh 389u IPv4
1020083623 TCP
victoria.ebuddy.com:51989->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 390u IPv4
1020083626 TCP
victoria.ebuddy.com:51990->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 391u IPv4
1020083637 TCP
victoria.ebuddy.com:51993->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 392u IPv4
1020083641 TCP
victoria.ebuddy.com:51994->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 393u IPv4
1020083668 TCP
victoria.ebuddy.com:52005->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 394u IPv4
1020083675 TCP
victoria.ebuddy.com:52008->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 395u IPv4
1020083678 TCP
victoria.ebuddy.com:52009->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 396u IPv4
1020083690 TCP
victoria.ebuddy.com:52014->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 397u IPv4
1020083693 TCP
victoria.ebuddy.com:52015->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 398u IPv4
1020083703 TCP
victoria.ebuddy.com:52019->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 399u IPv4
1020083706 TCP
victoria.ebuddy.com:52020->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 400u IPv4
1020083713 TCP
victoria.ebuddy.com:52023->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 401u IPv4
1020083716 TCP
victoria.ebuddy.com:52024->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 402u IPv4
1020083728 TCP
victoria.ebuddy.com:52029->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 403u IPv4
1020083731 TCP
victoria.ebuddy.com:52030->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 404u IPv4
1020083740 TCP
victoria.ebuddy.com:52034->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 405u IPv4
1020083743 TCP
victoria.ebuddy.com:52035->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 406u IPv4
1020083753 TCP
victoria.ebuddy.com:52039->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 407u IPv4
1020083756 TCP
victoria.ebuddy.com:52040->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 408u IPv4
1020083769 TCP
victoria.ebuddy.com:52045->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 409u IPv4
1020083772 TCP
victoria.ebuddy.com:52046->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 410u IPv4
1020083781 TCP
victoria.ebuddy.com:52050->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 411u IPv4
1020083784 TCP
victoria.ebuddy.com:52051->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 412u IPv4
1020083792 TCP
victoria.ebuddy.com:52054->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 413u IPv4
1020083795 TCP
victoria.ebuddy.com:52055->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 414u IPv4
1020083808 TCP
victoria.ebuddy.com:52060->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 415u IPv4
1020083811 TCP
victoria.ebuddy.com:52061->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 416u IPv4
1020083820 TCP
victoria.ebuddy.com:52065->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 417u IPv4
1020083823 TCP
victoria.ebuddy.com:52066->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 418u IPv4
1020083831 TCP
victoria.ebuddy.com:52069->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 419u IPv4
1020083834 TCP
victoria.ebuddy.com:52070->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 420u IPv4
1020083846 TCP
victoria.ebuddy.com:52075->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 421u IPv4
1020083849 TCP
victoria.ebuddy.com:52076->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 422u IPv4
1020083865 TCP
victoria.ebuddy.com:52083->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 423u IPv4
1020083862 TCP
victoria.ebuddy.com:52082->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 424u IPv4
1020083877 TCP
victoria.ebuddy.com:52088->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 425u IPv4
1020083880 TCP
victoria.ebuddy.com:52089->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 426u IPv4
1020083888 TCP
victoria.ebuddy.com:52092->victoria.ebuddy.com:50010 (ESTABLISHED)
java 5888 dwh 427u IPv4
1020083896 TCP
vict

Re: Hive Server Leaking File Descriptors?

Zheng Shao 2010-02-16, 08:08

What's running at victoria.ebuddy.com:50010 ?

Zhemg

On Tue, Feb 16, 2010 at 12:05 AM, Bennie Schut <[EMAIL PROTECTED]> wrote:
> I think it's lines like this:
> java 16908 hadoop 160u IPv6 153354839 TCP
> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
> (ESTABLISHED)
>
> We have the same.
> Currently no queries running and many of those connections open. After
> running some queries a few more are added to the list.
>
> java 5888 dwh 389u IPv4 1020083623
> TCP victoria.ebuddy.com:51989->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 390u IPv4 1020083626
> TCP victoria.ebuddy.com:51990->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 391u IPv4 1020083637
> TCP victoria.ebuddy.com:51993->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 392u IPv4 1020083641
> TCP victoria.ebuddy.com:51994->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 393u IPv4 1020083668
> TCP victoria.ebuddy.com:52005->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 394u IPv4 1020083675
> TCP victoria.ebuddy.com:52008->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 395u IPv4 1020083678
> TCP victoria.ebuddy.com:52009->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 396u IPv4 1020083690
> TCP victoria.ebuddy.com:52014->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 397u IPv4 1020083693
> TCP victoria.ebuddy.com:52015->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 398u IPv4 1020083703
> TCP victoria.ebuddy.com:52019->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 399u IPv4 1020083706
> TCP victoria.ebuddy.com:52020->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 400u IPv4 1020083713
> TCP victoria.ebuddy.com:52023->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 401u IPv4 1020083716
> TCP victoria.ebuddy.com:52024->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 402u IPv4 1020083728
> TCP victoria.ebuddy.com:52029->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 403u IPv4 1020083731
> TCP victoria.ebuddy.com:52030->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 404u IPv4 1020083740
> TCP victoria.ebuddy.com:52034->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 405u IPv4 1020083743
> TCP victoria.ebuddy.com:52035->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 406u IPv4 1020083753
> TCP victoria.ebuddy.com:52039->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 407u IPv4 1020083756
> TCP victoria.ebuddy.com:52040->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 408u IPv4 1020083769
> TCP victoria.ebuddy.com:52045->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 409u IPv4 1020083772
> TCP victoria.ebuddy.com:52046->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 410u IPv4 1020083781
> TCP victoria.ebuddy.com:52050->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 411u IPv4 1020083784
> TCP victoria.ebuddy.com:52051->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 412u IPv4 1020083792
> TCP victoria.ebuddy.com:52054->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 413u IPv4 1020083795
> TCP victoria.ebuddy.com:52055->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 414u IPv4 1020083808
> TCP victoria.ebuddy.com:52060->victoria.ebuddy.com:50010 (ESTABLISHED)
> java 5888 dwh 415u IPv4 1020083811
> TCP victoria.ebuddy.com:52061->victoria.ebuddy.com:50010 (ESTABLISHED)

Yours,
Zheng

Re: Hive Server Leaking File Descriptors?

Bennie Schut 2010-02-16, 08:13

That's the datanode port. So Hdfs. Any trick to see what is opened on
hdfs? like an lsof for hdfs :)

Zheng Shao wrote:
> What's running at victoria.ebuddy.com:50010 ?
>
> Zhemg
>
> On Tue, Feb 16, 2010 at 12:05 AM, Bennie Schut <[EMAIL PROTECTED]> wrote:
>
>> I think it's lines like this:
>> java 16908 hadoop 160u IPv6 153354839 TCP
>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>> (ESTABLISHED)
>>
>> We have the same.
>> Currently no queries running and many of those connections open. After
>> running some queries a few more are added to the list.
>>
>> java 5888 dwh 389u IPv4 1020083623
>> TCP victoria.ebuddy.com:51989->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 390u IPv4 1020083626
>> TCP victoria.ebuddy.com:51990->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 391u IPv4 1020083637
>> TCP victoria.ebuddy.com:51993->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 392u IPv4 1020083641
>> TCP victoria.ebuddy.com:51994->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 393u IPv4 1020083668
>> TCP victoria.ebuddy.com:52005->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 394u IPv4 1020083675
>> TCP victoria.ebuddy.com:52008->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 395u IPv4 1020083678
>> TCP victoria.ebuddy.com:52009->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 396u IPv4 1020083690
>> TCP victoria.ebuddy.com:52014->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 397u IPv4 1020083693
>> TCP victoria.ebuddy.com:52015->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 398u IPv4 1020083703
>> TCP victoria.ebuddy.com:52019->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 399u IPv4 1020083706
>> TCP victoria.ebuddy.com:52020->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 400u IPv4 1020083713
>> TCP victoria.ebuddy.com:52023->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 401u IPv4 1020083716
>> TCP victoria.ebuddy.com:52024->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 402u IPv4 1020083728
>> TCP victoria.ebuddy.com:52029->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 403u IPv4 1020083731
>> TCP victoria.ebuddy.com:52030->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 404u IPv4 1020083740
>> TCP victoria.ebuddy.com:52034->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 405u IPv4 1020083743
>> TCP victoria.ebuddy.com:52035->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 406u IPv4 1020083753
>> TCP victoria.ebuddy.com:52039->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 407u IPv4 1020083756
>> TCP victoria.ebuddy.com:52040->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 408u IPv4 1020083769
>> TCP victoria.ebuddy.com:52045->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 409u IPv4 1020083772
>> TCP victoria.ebuddy.com:52046->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 410u IPv4 1020083781
>> TCP victoria.ebuddy.com:52050->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 411u IPv4 1020083784
>> TCP victoria.ebuddy.com:52051->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 412u IPv4 1020083792
>> TCP victoria.ebuddy.com:52054->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 413u IPv4 1020083795
>> TCP victoria.ebuddy.com:52055->victoria.ebuddy.com:50010 (ESTABLISHED)
>> java 5888 dwh 414u IPv4 1020083808

Re: Hive Server Leaking File Descriptors?

Zheng Shao 2010-02-16, 08:22

It seems that Hive server is leaking hdfs handles.
Not sure if hadoop has a command to show open hdfs handles.

Can you also do a jstack of the Hive Server process?
First, "ps -elf | grep java" to find the hive server process id, and
the jdk directory
Second, "<jdk directory>/bin/jstack <pid>".

This will tell us if there are clients of the Hive Server that didn't
close their connections yet.

Zheng

On Tue, Feb 16, 2010 at 12:13 AM, Bennie Schut <[EMAIL PROTECTED]> wrote:
> That's the datanode port. So Hdfs. Any trick to see what is opened on hdfs?
> like an lsof for hdfs :)
>
> Zheng Shao wrote:
>>
>> What's running at victoria.ebuddy.com:50010 ?
>>
>> Zhemg
>>
>> On Tue, Feb 16, 2010 at 12:05 AM, Bennie Schut <[EMAIL PROTECTED]> wrote:
>>
>>>
>>> I think it's lines like this:
>>> java 16908 hadoop 160u IPv6 153354839 TCP
>>>
>>> hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010
>>> (ESTABLISHED)
>>>
>>> We have the same.
>>> Currently no queries running and many of those connections open. After
>>> running some queries a few more are added to the list.
>>>
>>> java 5888 dwh 389u IPv4 1020083623
>>> TCP victoria.ebuddy.com:51989->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 390u IPv4 1020083626
>>> TCP victoria.ebuddy.com:51990->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 391u IPv4 1020083637
>>> TCP victoria.ebuddy.com:51993->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 392u IPv4 1020083641
>>> TCP victoria.ebuddy.com:51994->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 393u IPv4 1020083668
>>> TCP victoria.ebuddy.com:52005->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 394u IPv4 1020083675
>>> TCP victoria.ebuddy.com:52008->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 395u IPv4 1020083678
>>> TCP victoria.ebuddy.com:52009->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 396u IPv4 1020083690
>>> TCP victoria.ebuddy.com:52014->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 397u IPv4 1020083693
>>> TCP victoria.ebuddy.com:52015->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 398u IPv4 1020083703
>>> TCP victoria.ebuddy.com:52019->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 399u IPv4 1020083706
>>> TCP victoria.ebuddy.com:52020->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 400u IPv4 1020083713
>>> TCP victoria.ebuddy.com:52023->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 401u IPv4 1020083716
>>> TCP victoria.ebuddy.com:52024->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 402u IPv4 1020083728
>>> TCP victoria.ebuddy.com:52029->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 403u IPv4 1020083731
>>> TCP victoria.ebuddy.com:52030->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 404u IPv4 1020083740
>>> TCP victoria.ebuddy.com:52034->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 405u IPv4 1020083743
>>> TCP victoria.ebuddy.com:52035->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 406u IPv4 1020083753
>>> TCP victoria.ebuddy.com:52039->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 407u IPv4 1020083756
>>> TCP victoria.ebuddy.com:52040->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 408u IPv4 1020083769
>>> TCP victoria.ebuddy.com:52045->victoria.ebuddy.com:50010 (ESTABLISHED)
>>> java 5888 dwh 409u IPv4 1020083772
>>> TCP victoria.ebuddy.com:52046->victoria.ebuddy.com:50010 (ESTABLISHED)
>>
Yours,
Zheng

Re: Hive Server Leaking File Descriptors?

Bennie Schut 2010-02-16, 08:32

jstack on the Hive process:

2010-02-16
09:30:47

Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed
mode):
"Attach Listener" daemon prio=10 tid=0x00007fbe9527f800 nid=0x4a48
waiting on condition [0x0000000000000000]
   java.lang.Thread.State:
RUNNABLE
"pool-1-thread-5" prio=10 tid=0x00007fbe95efd000 nid=0x6bab waiting on
condition [0x000000004238d000]
   java.lang.Thread.State: WAITING
(parking)
   at sun.misc.Unsafe.$$YJP$$park(Native
Method)
   - parking to wait for <0x00007fbea529a0d8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
   at
sun.misc.Unsafe.park(Unsafe.java)

   at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

   at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)

   at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)

   at
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)

   at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

   at
java.lang.Thread.run(Thread.java:619)
"pool-1-thread-4" prio=10 tid=0x00007fbe963e0000 nid=0x6b87 waiting on
condition [0x0000000042690000]
   java.lang.Thread.State: WAITING
(parking)
   at sun.misc.Unsafe.$$YJP$$park(Native
Method)
   - parking to wait for <0x00007fbea529a0d8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
   at
sun.misc.Unsafe.park(Unsafe.java)

   at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

   at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)

   at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)

   at
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)

   at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

   at
java.lang.Thread.run(Thread.java:619)
"LeaseChecker" daemon prio=10 tid=0x00007fbe9653c800 nid=0x1920 waiting
on condition [0x0000000042791000]
   java.lang.Thread.State: TIMED_WAITING
(sleeping)
   at java.lang.Thread.$$YJP$$sleep(Native
Method)
   at
java.lang.Thread.sleep(Thread.java)

   at
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1066)

   at
java.lang.Thread.run(Thread.java:619)
"pool-1-thread-3" prio=10 tid=0x00007fbe963ac000 nid=0x18a9 waiting on
condition [0x000000004258f000]
   java.lang.Thread.State: WAITING
(parking)
   at sun.misc.Unsafe.$$YJP$$park(Native
Method)
   - parking to wait for <0x00007fbea529a0d8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
   at
sun.misc.Unsafe.park(Unsafe.java)

   at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

   at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)

   at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)

   at
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)

   at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

   at
java.lang.Thread.run(Thread.java:619)
"pool-1-thread-2" prio=10 tid=0x00007fbe95fbf800 nid=0x18a5 waiting on
condition [0x000000004248e000]
   java.lang.Thread.State: WAITING
(parking)
   at sun.misc.Unsafe.$$YJP$$park(Native
Method)
   - parking to wait for <0x00007fbea529a0d8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
   at
sun.misc.Unsafe.park(Unsafe.java)

   at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

   at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)

   at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)

   at

Re: Hive Server Leaking File Descriptors?

Zheng Shao 2010-02-16, 09:05

Thanks for the quick reply. It seems the number of threads are normal.

Can you do "jmap -histo:live" as well to find out the number of
DFSClient, FileSystem, etc?
On 2/16/10, Bennie Schut <[EMAIL PROTECTED]> wrote:
> jstack on the Hive process:
>
> 2010-02-16
> 09:30:47
>
>
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed
> mode):
>
>
>
> "Attach Listener" daemon prio=10 tid=0x00007fbe9527f800 nid=0x4a48
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State:
> RUNNABLE
>
>
>
> "pool-1-thread-5" prio=10 tid=0x00007fbe95efd000 nid=0x6bab waiting on
> condition [0x000000004238d000]
> java.lang.Thread.State: WAITING
> (parking)
> at sun.misc.Unsafe.$$YJP$$park(Native
> Method)
> - parking to wait for <0x00007fbea529a0d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> sun.misc.Unsafe.park(Unsafe.java)
>
>
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>
>
> at
> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
>
> at
> java.lang.Thread.run(Thread.java:619)
>
>
>
> "pool-1-thread-4" prio=10 tid=0x00007fbe963e0000 nid=0x6b87 waiting on
> condition [0x0000000042690000]
> java.lang.Thread.State: WAITING
> (parking)
> at sun.misc.Unsafe.$$YJP$$park(Native
> Method)
> - parking to wait for <0x00007fbea529a0d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> sun.misc.Unsafe.park(Unsafe.java)
>
>
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>
>
> at
> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
>
> at
> java.lang.Thread.run(Thread.java:619)
>
>
>
> "LeaseChecker" daemon prio=10 tid=0x00007fbe9653c800 nid=0x1920 waiting
> on condition [0x0000000042791000]
> java.lang.Thread.State: TIMED_WAITING
> (sleeping)
> at java.lang.Thread.$$YJP$$sleep(Native
> Method)
> at
> java.lang.Thread.sleep(Thread.java)
>
>
> at
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1066)
>
>
> at
> java.lang.Thread.run(Thread.java:619)
>
>
>
> "pool-1-thread-3" prio=10 tid=0x00007fbe963ac000 nid=0x18a9 waiting on
> condition [0x000000004258f000]
> java.lang.Thread.State: WAITING
> (parking)
> at sun.misc.Unsafe.$$YJP$$park(Native
> Method)
> - parking to wait for <0x00007fbea529a0d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> sun.misc.Unsafe.park(Unsafe.java)
>
>
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>
>
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>
>
> at
> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>
>
> at
> java.lang.Thread.run(Thread.java:619)

Sent from my mobile device

Yours,
Zheng

Re: Hive Server Leaking File Descriptors?

Bennie Schut 2010-02-16, 09:36

That's 133k of data so perhaps this is enough?:

627: 1 112
org.apache.hadoop.fs.FileSystem$ClientFinalizer
723: 1 80 org.apache.hadoop.hdfs.DFSClient
882: 1 48 org.apache.hadoop.fs.LocalFileSystem
883: 2 48
org.apache.hadoop.fs.FileSystem$Cache$Key
934: 2 48
org.apache.hadoop.fs.FileSystem$Statistics
944: 1 48
org.apache.hadoop.hdfs.DistributedFileSystem
1082: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
1643: 1 16 org.apache.hadoop.fs.FileSystem$Cache

After a couple more queries:

671: 1 112
org.apache.hadoop.fs.FileSystem$ClientFinalizer
772: 1 80 org.apache.hadoop.hdfs.DFSClient
930: 1 48 org.apache.hadoop.fs.LocalFileSystem
931: 2 48
org.apache.hadoop.fs.FileSystem$Cache$Key
981: 2 48
org.apache.hadoop.fs.FileSystem$Statistics
991: 1 48
org.apache.hadoop.hdfs.DistributedFileSystem
1132: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache

some more stuff
cat jmap.txt | grep hadoop.fs
535: 8 192
org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream
539: 8 192 org.apache.hadoop.fs.permission.FsAction
611: 8 128 org.apache.hadoop.fs.Path
671: 1 112
org.apache.hadoop.fs.FileSystem$ClientFinalizer
721: 4 96
org.apache.hadoop.fs.permission.FsPermission$2
760: 2 96
[Lorg.apache.hadoop.fs.permission.FsAction;
930: 1 48 org.apache.hadoop.fs.LocalFileSystem
931: 2 48
org.apache.hadoop.fs.FileSystem$Cache$Key
981: 2 48
org.apache.hadoop.fs.FileSystem$Statistics
1132: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
1213: 1 24
org.apache.hadoop.fs.permission.FsPermission
1347: 1 16 org.apache.hadoop.fs.FileSystem$1
1413: 1 16
org.apache.hadoop.fs.ChecksumFileSystem$1
1523: 1 16
org.apache.hadoop.fs.permission.FsPermission$1
1702: 1 16 org.apache.hadoop.fs.BlockLocation$1
1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache

Currently:
lsof | grep "50010 (ESTABLISHED)" | wc -l
453

Zheng Shao wrote:
> Thanks for the quick reply. It seems the number of threads are normal.
>
> Can you do "jmap -histo:live" as well to find out the number of
> DFSClient, FileSystem, etc?
>
>
> On 2/16/10, Bennie Schut <[EMAIL PROTECTED]> wrote:
>
>> jstack on the Hive process:
>>
>> 2010-02-16
>> 09:30:47
>>
>>
>>
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed
>> mode):
>>
>>
>>
>> "Attach Listener" daemon prio=10 tid=0x00007fbe9527f800 nid=0x4a48
>> waiting on condition [0x0000000000000000]
>> java.lang.Thread.State:
>> RUNNABLE
>>
>>
>>
>> "pool-1-thread-5" prio=10 tid=0x00007fbe95efd000 nid=0x6bab waiting on
>> condition [0x000000004238d000]
>> java.lang.Thread.State: WAITING
>> (parking)
>> at sun.misc.Unsafe.$$YJP$$park(Native
>> Method)
>> - parking to wait for <0x00007fbea529a0d8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>> at
>> sun.misc.Unsafe.park(Unsafe.java)
>>
>>
>> at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>
>>
>> at
>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)

Re: Hive Server Leaking File Descriptors?

Zheng Shao 2010-02-16, 19:10

These all seems to be normal.
Can you attach the 133k file as an attachment? (At least all lines
with org.apache.hadoop.*, and java.*)

Zheng

On Tue, Feb 16, 2010 at 1:36 AM, Bennie Schut <[EMAIL PROTECTED]> wrote:
> That's 133k of data so perhaps this is enough?:
>
> 627: 1 112
> org.apache.hadoop.fs.FileSystem$ClientFinalizer
> 723: 1 80 org.apache.hadoop.hdfs.DFSClient
> 882: 1 48 org.apache.hadoop.fs.LocalFileSystem
> 883: 2 48 org.apache.hadoop.fs.FileSystem$Cache$Key
> 934: 2 48
> org.apache.hadoop.fs.FileSystem$Statistics
> 944: 1 48
> org.apache.hadoop.hdfs.DistributedFileSystem
> 1082: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
> 1643: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>
> After a couple more queries:
>
> 671: 1 112
> org.apache.hadoop.fs.FileSystem$ClientFinalizer
> 772: 1 80 org.apache.hadoop.hdfs.DFSClient
> 930: 1 48 org.apache.hadoop.fs.LocalFileSystem
> 931: 2 48 org.apache.hadoop.fs.FileSystem$Cache$Key
> 981: 2 48
> org.apache.hadoop.fs.FileSystem$Statistics
> 991: 1 48
> org.apache.hadoop.hdfs.DistributedFileSystem
> 1132: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
> 1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>
> some more stuff
> cat jmap.txt | grep hadoop.fs
> 535: 8 192
> org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream
> 539: 8 192 org.apache.hadoop.fs.permission.FsAction
> 611: 8 128 org.apache.hadoop.fs.Path
> 671: 1 112
> org.apache.hadoop.fs.FileSystem$ClientFinalizer
> 721: 4 96
> org.apache.hadoop.fs.permission.FsPermission$2
> 760: 2 96
> [Lorg.apache.hadoop.fs.permission.FsAction;
> 930: 1 48 org.apache.hadoop.fs.LocalFileSystem
> 931: 2 48 org.apache.hadoop.fs.FileSystem$Cache$Key
> 981: 2 48
> org.apache.hadoop.fs.FileSystem$Statistics
> 1132: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
> 1213: 1 24
> org.apache.hadoop.fs.permission.FsPermission
> 1347: 1 16 org.apache.hadoop.fs.FileSystem$1
> 1413: 1 16
> org.apache.hadoop.fs.ChecksumFileSystem$1
> 1523: 1 16
> org.apache.hadoop.fs.permission.FsPermission$1
> 1702: 1 16 org.apache.hadoop.fs.BlockLocation$1
> 1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>
>
>
> Currently:
> lsof | grep "50010 (ESTABLISHED)" | wc -l
> 453
>
> Zheng Shao wrote:
>>
>> Thanks for the quick reply. It seems the number of threads are normal.
>>
>> Can you do "jmap -histo:live" as well to find out the number of
>> DFSClient, FileSystem, etc?
>>
>>
>> On 2/16/10, Bennie Schut <[EMAIL PROTECTED]> wrote:
>>
>>>
>>> jstack on the Hive process:
>>>
>>> 2010-02-16
>>> 09:30:47
>>>
>>>
>>>
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed
>>> mode):
>>>
>>>
>>>
>>> "Attach Listener" daemon prio=10 tid=0x00007fbe9527f800 nid=0x4a48
>>> waiting on condition [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>>
>>>
>>> "pool-1-thread-5" prio=10 tid=0x00007fbe95efd000 nid=0x6bab waiting on
>>> condition [0x000000004238d000]
>>> java.lang.Thread.State: WAITING
>>> (parking)
>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>> Method)
>>> - parking to wait for <0x00007fbea529a0d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>> at
>>> sun.misc.Unsafe.park(Unsafe.java)
>>>
>>>
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

Yours,
Zheng

Re: Hive Server Leaking File Descriptors?

Bennie Schut 2010-02-18, 14:27

I've tried to look into it a bit more and it seems to happen on "load
data inpath"
LOAD DATA INPATH '/user/hive/warehouse/chatsessions_2010-02-01.csv' INTO
TABLE chatsessions_load;

And not on things like
select count(1) from chatsessions_load;

Bennie Schut wrote:
> That's 133k of data so perhaps this is enough?:
>
> 627: 1 112
> org.apache.hadoop.fs.FileSystem$ClientFinalizer
> 723: 1 80 org.apache.hadoop.hdfs.DFSClient
> 882: 1 48 org.apache.hadoop.fs.LocalFileSystem
> 883: 2 48
> org.apache.hadoop.fs.FileSystem$Cache$Key
> 934: 2 48
> org.apache.hadoop.fs.FileSystem$Statistics
> 944: 1 48
> org.apache.hadoop.hdfs.DistributedFileSystem
> 1082: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
> 1643: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>
> After a couple more queries:
>
> 671: 1 112
> org.apache.hadoop.fs.FileSystem$ClientFinalizer
> 772: 1 80 org.apache.hadoop.hdfs.DFSClient
> 930: 1 48 org.apache.hadoop.fs.LocalFileSystem
> 931: 2 48
> org.apache.hadoop.fs.FileSystem$Cache$Key
> 981: 2 48
> org.apache.hadoop.fs.FileSystem$Statistics
> 991: 1 48
> org.apache.hadoop.hdfs.DistributedFileSystem
> 1132: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
> 1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>
> some more stuff
> cat jmap.txt | grep hadoop.fs
> 535: 8 192
> org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream
> 539: 8 192 org.apache.hadoop.fs.permission.FsAction
> 611: 8 128 org.apache.hadoop.fs.Path
> 671: 1 112
> org.apache.hadoop.fs.FileSystem$ClientFinalizer
> 721: 4 96
> org.apache.hadoop.fs.permission.FsPermission$2
> 760: 2 96
> [Lorg.apache.hadoop.fs.permission.FsAction;
> 930: 1 48 org.apache.hadoop.fs.LocalFileSystem
> 931: 2 48
> org.apache.hadoop.fs.FileSystem$Cache$Key
> 981: 2 48
> org.apache.hadoop.fs.FileSystem$Statistics
> 1132: 1 32 org.apache.hadoop.fs.RawLocalFileSystem
> 1213: 1 24
> org.apache.hadoop.fs.permission.FsPermission
> 1347: 1 16 org.apache.hadoop.fs.FileSystem$1
> 1413: 1 16
> org.apache.hadoop.fs.ChecksumFileSystem$1
> 1523: 1 16
> org.apache.hadoop.fs.permission.FsPermission$1
> 1702: 1 16 org.apache.hadoop.fs.BlockLocation$1
> 1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>
>
>
> Currently:
> lsof | grep "50010 (ESTABLISHED)" | wc -l
> 453
>
> Zheng Shao wrote:
>
>> Thanks for the quick reply. It seems the number of threads are normal.
>>
>> Can you do "jmap -histo:live" as well to find out the number of
>> DFSClient, FileSystem, etc?
>>
>>
>> On 2/16/10, Bennie Schut <[EMAIL PROTECTED]> wrote:
>>
>>
>>> jstack on the Hive process:
>>>
>>> 2010-02-16
>>> 09:30:47
>>>
>>>
>>>
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed
>>> mode):
>>>
>>>
>>>
>>> "Attach Listener" daemon prio=10 tid=0x00007fbe9527f800 nid=0x4a48
>>> waiting on condition [0x0000000000000000]
>>> java.lang.Thread.State:
>>> RUNNABLE
>>>
>>>
>>>
>>> "pool-1-thread-5" prio=10 tid=0x00007fbe95efd000 nid=0x6bab waiting on
>>> condition [0x000000004238d000]
>>> java.lang.Thread.State: WAITING
>>> (parking)
>>> at sun.misc.Unsafe.$$YJP$$park(Native
>>> Method)
>>> - parking to wait for <0x00007fbea529a0d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>> at

Re: Hive Server Leaking File Descriptors?

Zheng Shao 2010-02-18, 20:28

Bennie,

Thanks for the detailed information! I think we've found the bug.
I just opened this:
https://issues.apache.org/jira/browse/MAPREDUCE-1504

Hive does check of the file format (to be consistent with the file
format declared in the table) when users load data files into Hive.
In order to verify that a file is a text file, Hive uses
SequenceFile.Reader class to open the file. If an IOException is
thrown, we know it's not a SequenceFile.

However, when an IOException is thrown, the
SequenceFile.Reader.close() is not called. I believe that is leaking
resources.
Please take a look at the JIRA above and put in any comments.

Zheng

On Thu, Feb 18, 2010 at 6:27 AM, Bennie Schut <[EMAIL PROTECTED]> wrote:
> I've tried to look into it a bit more and it seems to happen on "load data
> inpath"
> LOAD DATA INPATH '/user/hive/warehouse/chatsessions_2010-02-01.csv' INTO
> TABLE chatsessions_load;
>
> And not on things like
> select count(1) from chatsessions_load;
>
>
>
> Bennie Schut wrote:
>>
>> That's 133k of data so perhaps this is enough?:
>>
>> 627: 1 112
>> org.apache.hadoop.fs.FileSystem$ClientFinalizer
>> 723: 1 80 org.apache.hadoop.hdfs.DFSClient
>> 882: 1 48 org.apache.hadoop.fs.LocalFileSystem
>> 883: 2 48
>> org.apache.hadoop.fs.FileSystem$Cache$Key
>> 934: 2 48
>> org.apache.hadoop.fs.FileSystem$Statistics
>> 944: 1 48
>> org.apache.hadoop.hdfs.DistributedFileSystem
>> 1082: 1 32
>> org.apache.hadoop.fs.RawLocalFileSystem
>> 1643: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>>
>> After a couple more queries:
>>
>> 671: 1 112
>> org.apache.hadoop.fs.FileSystem$ClientFinalizer
>> 772: 1 80 org.apache.hadoop.hdfs.DFSClient
>> 930: 1 48 org.apache.hadoop.fs.LocalFileSystem
>> 931: 2 48
>> org.apache.hadoop.fs.FileSystem$Cache$Key
>> 981: 2 48
>> org.apache.hadoop.fs.FileSystem$Statistics
>> 991: 1 48
>> org.apache.hadoop.hdfs.DistributedFileSystem
>> 1132: 1 32
>> org.apache.hadoop.fs.RawLocalFileSystem
>> 1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>>
>> some more stuff
>> cat jmap.txt | grep hadoop.fs
>> 535: 8 192
>> org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream
>> 539: 8 192
>> org.apache.hadoop.fs.permission.FsAction
>> 611: 8 128 org.apache.hadoop.fs.Path
>> 671: 1 112
>> org.apache.hadoop.fs.FileSystem$ClientFinalizer
>> 721: 4 96
>> org.apache.hadoop.fs.permission.FsPermission$2
>> 760: 2 96
>> [Lorg.apache.hadoop.fs.permission.FsAction;
>> 930: 1 48 org.apache.hadoop.fs.LocalFileSystem
>> 931: 2 48
>> org.apache.hadoop.fs.FileSystem$Cache$Key
>> 981: 2 48
>> org.apache.hadoop.fs.FileSystem$Statistics
>> 1132: 1 32
>> org.apache.hadoop.fs.RawLocalFileSystem
>> 1213: 1 24
>> org.apache.hadoop.fs.permission.FsPermission
>> 1347: 1 16 org.apache.hadoop.fs.FileSystem$1
>> 1413: 1 16
>> org.apache.hadoop.fs.ChecksumFileSystem$1
>> 1523: 1 16
>> org.apache.hadoop.fs.permission.FsPermission$1
>> 1702: 1 16 org.apache.hadoop.fs.BlockLocation$1
>> 1743: 1 16 org.apache.hadoop.fs.FileSystem$Cache
>>
>>
>>
>> Currently:
>> lsof | grep "50010 (ESTABLISHED)" | wc -l
>> 453
>>
>> Zheng Shao wrote:
>>
>>>
>>> Thanks for the quick reply. It seems the number of threads are normal.
>>>
>>> Can you do "jmap -histo:live" as well to find out the number of

Yours,
Zheng

Re: Hive Server Leaking File Descriptors?

Zheng Shao 2010-02-18, 23:45

This is actually a bug in MAPREDUCE-1504, but we will try to find a workaround.
https://issues.apache.org/jira/browse/HIVE-1181

Given that release 0.5.0 is much wanted right now, I don't think we
want to wait purely for 0.5.0 since the ultimate fix should come from
Hadoop.
We will definitely get HIVE-1181 for branch 0.5.

Zheng

---------- Forwarded message ----------
From: Andy Kent <[EMAIL PROTECTED]>
Date: Thu, Feb 18, 2010 at 3:17 PM
Subject: Re: Hive Server Leaking File Descriptors?
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>

On 18 Feb 2010, at 20:29, "Zheng Shao" <[EMAIL PROTECTED]> wrote:

>> I've tried to look into it a bit more and it seems to happen on
>> "load data
>> inpath"

This is inline with what we have been seeing as we do around 200 load
data statements per day and leak approx the same number of file
descriptors.

Is there any chance this fix will make it into the 0.5 release?

--
Yours,
Zheng

Re: Hive Server Leaking File Descriptors?

Bennie Schut 2010-02-19, 07:47

That's some great news. Thanks.

Zheng Shao wrote:
> This is actually a bug in MAPREDUCE-1504, but we will try to find a workaround.
> https://issues.apache.org/jira/browse/HIVE-1181
>
> Given that release 0.5.0 is much wanted right now, I don't think we
> want to wait purely for 0.5.0 since the ultimate fix should come from
> Hadoop.
> We will definitely get HIVE-1181 for branch 0.5.
>
> Zheng
>
> ---------- Forwarded message ----------
> From: Andy Kent <[EMAIL PROTECTED]>
> Date: Thu, Feb 18, 2010 at 3:17 PM
> Subject: Re: Hive Server Leaking File Descriptors?
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>
>
>
> On 18 Feb 2010, at 20:29, "Zheng Shao" <[EMAIL PROTECTED]> wrote:
>
>
>>> I've tried to look into it a bit more and it seems to happen on
>>> "load data
>>> inpath"
>>>
>
> This is inline with what we have been seeing as we do around 200 load
> data statements per day and leak approx the same number of file
> descriptors.
>
> Is there any chance this fix will make it into the 0.5 release?
>
>
>
>

RE: Hive Server Leaking File Descriptors?

Andy Kent 2010-02-19, 14:07

This has made my day.

Thanks for working through this Bennie and Zheng.

Andy.
________________________________________
From: Bennie Schut [[EMAIL PROTECTED]]
Sent: 19 February 2010 07:47
To: [EMAIL PROTECTED]
Subject: Re: Hive Server Leaking File Descriptors?

That's some great news. Thanks.

Zheng Shao wrote:
> This is actually a bug in MAPREDUCE-1504, but we will try to find a workaround.
> https://issues.apache.org/jira/browse/HIVE-1181
>
> Given that release 0.5.0 is much wanted right now, I don't think we
> want to wait purely for 0.5.0 since the ultimate fix should come from
> Hadoop.
> We will definitely get HIVE-1181 for branch 0.5.
>
> Zheng
>
> ---------- Forwarded message ----------
> From: Andy Kent <[EMAIL PROTECTED]>
> Date: Thu, Feb 18, 2010 at 3:17 PM
> Subject: Re: Hive Server Leaking File Descriptors?
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>
>
>
> On 18 Feb 2010, at 20:29, "Zheng Shao" <[EMAIL PROTECTED]> wrote:
>
>
>>> I've tried to look into it a bit more and it seems to happen on
>>> "load data
>>> inpath"
>>>
>
> This is inline with what we have been seeing as we do around 200 load
> data statements per day and leak approx the same number of file
> descriptors.
>
> Is there any chance this fix will make it into the 0.5 release?
>
>
>
>

Re: Hive Server Leaking File Descriptors?

Zheng Shao 2010-02-20, 02:38

https://issues.apache.org/jira/browse/HIVE-1181 is committed.
Thanks to Yonqqiang.

We need to "-hiveconf hive.fileformat.check=false" when starting
HiveServer to get rid of the extra connections.

We still need to fix MAPREDUCE-1504 so that we can re-enable fileformat check.

Zheng

On Thu, Feb 18, 2010 at 11:47 PM, Bennie Schut <[EMAIL PROTECTED]> wrote:
> That's some great news. Thanks.
>
> Zheng Shao wrote:
>>
>> This is actually a bug in MAPREDUCE-1504, but we will try to find a
>> workaround.
>> https://issues.apache.org/jira/browse/HIVE-1181
>>
>> Given that release 0.5.0 is much wanted right now, I don't think we
>> want to wait purely for 0.5.0 since the ultimate fix should come from
>> Hadoop.
>> We will definitely get HIVE-1181 for branch 0.5.
>>
>> Zheng
>>
>> ---------- Forwarded message ----------
>> From: Andy Kent <[EMAIL PROTECTED]>
>> Date: Thu, Feb 18, 2010 at 3:17 PM
>> Subject: Re: Hive Server Leaking File Descriptors?
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>
>>
>>
>> On 18 Feb 2010, at 20:29, "Zheng Shao" <[EMAIL PROTECTED]> wrote:
>>
>>
>>>>
>>>> I've tried to look into it a bit more and it seems to happen on
>>>> "load data
>>>> inpath"
>>>>
>>
>> This is inline with what we have been seeing as we do around 200 load
>> data statements per day and leak approx the same number of file
>> descriptors.
>>
>> Is there any chance this fix will make it into the 0.5 release?
>>
>>
>>
>>
>
>

--
Yours,
Zheng

Re: Hive Server Leaking File Descriptors?

Bennie Schut 2010-02-23, 09:02

fyi.. just tested this and I see no more leaks. Thanks.

Zheng Shao wrote:
> https://issues.apache.org/jira/browse/HIVE-1181 is committed.
> Thanks to Yonqqiang.
>
> We need to "-hiveconf hive.fileformat.check=false" when starting
> HiveServer to get rid of the extra connections.
>
> We still need to fix MAPREDUCE-1504 so that we can re-enable fileformat check.
>
> Zheng
>
> On Thu, Feb 18, 2010 at 11:47 PM, Bennie Schut <[EMAIL PROTECTED]> wrote:
>
>> That's some great news. Thanks.
>>
>> Zheng Shao wrote:
>>
>>> This is actually a bug in MAPREDUCE-1504, but we will try to find a
>>> workaround.
>>> https://issues.apache.org/jira/browse/HIVE-1181
>>>
>>> Given that release 0.5.0 is much wanted right now, I don't think we
>>> want to wait purely for 0.5.0 since the ultimate fix should come from
>>> Hadoop.
>>> We will definitely get HIVE-1181 for branch 0.5.
>>>
>>> Zheng
>>>
>>> ---------- Forwarded message ----------
>>> From: Andy Kent <[EMAIL PROTECTED]>
>>> Date: Thu, Feb 18, 2010 at 3:17 PM
>>> Subject: Re: Hive Server Leaking File Descriptors?
>>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>>
>>>
>>>
>>> On 18 Feb 2010, at 20:29, "Zheng Shao" <[EMAIL PROTECTED]> wrote:
>>>
>>>
>>>
>>>>> I've tried to look into it a bit more and it seems to happen on
>>>>> "load data
>>>>> inpath"
>>>>>
>>>>>
>>> This is inline with what we have been seeing as we do around 200 load
>>> data statements per day and leak approx the same number of file
>>> descriptors.
>>>
>>> Is there any chance this fix will make it into the 0.5 release?
>>>
>>>
>>>
>>>
>>>
>>
>
>
>
>