Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Hive Server Leaking File Descriptors?


Copy link to this message
-
Re: Hive Server Leaking File Descriptors?
That's 133k of data so perhaps this is enough?:

 627:             1            112  
org.apache.hadoop.fs.FileSystem$ClientFinalizer
 723:             1             80  org.apache.hadoop.hdfs.DFSClient
 882:             1             48  org.apache.hadoop.fs.LocalFileSystem
 883:             2             48  
org.apache.hadoop.fs.FileSystem$Cache$Key
 934:             2             48  
org.apache.hadoop.fs.FileSystem$Statistics
 944:             1             48  
org.apache.hadoop.hdfs.DistributedFileSystem
1082:             1             32  org.apache.hadoop.fs.RawLocalFileSystem
1643:             1             16  org.apache.hadoop.fs.FileSystem$Cache

After a couple more queries:

671:             1            112  
org.apache.hadoop.fs.FileSystem$ClientFinalizer
772:             1             80  org.apache.hadoop.hdfs.DFSClient
 930:             1             48  org.apache.hadoop.fs.LocalFileSystem
 931:             2             48  
org.apache.hadoop.fs.FileSystem$Cache$Key
981:             2             48  
org.apache.hadoop.fs.FileSystem$Statistics
 991:             1             48  
org.apache.hadoop.hdfs.DistributedFileSystem
1132:             1             32  org.apache.hadoop.fs.RawLocalFileSystem
1743:             1             16  org.apache.hadoop.fs.FileSystem$Cache

some more stuff
cat jmap.txt | grep hadoop.fs
 535:             8            192  
org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream
 539:             8            192  org.apache.hadoop.fs.permission.FsAction
 611:             8            128  org.apache.hadoop.fs.Path
 671:             1            112  
org.apache.hadoop.fs.FileSystem$ClientFinalizer
 721:             4             96  
org.apache.hadoop.fs.permission.FsPermission$2
 760:             2             96  
[Lorg.apache.hadoop.fs.permission.FsAction;
 930:             1             48  org.apache.hadoop.fs.LocalFileSystem
 931:             2             48  
org.apache.hadoop.fs.FileSystem$Cache$Key
 981:             2             48  
org.apache.hadoop.fs.FileSystem$Statistics
1132:             1             32  org.apache.hadoop.fs.RawLocalFileSystem
1213:             1             24  
org.apache.hadoop.fs.permission.FsPermission
1347:             1             16  org.apache.hadoop.fs.FileSystem$1
1413:             1             16  
org.apache.hadoop.fs.ChecksumFileSystem$1
1523:             1             16  
org.apache.hadoop.fs.permission.FsPermission$1
1702:             1             16  org.apache.hadoop.fs.BlockLocation$1
1743:             1             16  org.apache.hadoop.fs.FileSystem$Cache

Currently:
lsof | grep "50010 (ESTABLISHED)" | wc -l
453

Zheng Shao wrote:
> Thanks for the quick reply. It seems the number of threads are normal.
>
> Can you do "jmap -histo:live" as well to find out the number of
> DFSClient, FileSystem, etc?
>
>
> On 2/16/10, Bennie Schut <[EMAIL PROTECTED]> wrote:
>  
>> jstack on the Hive process:
>>
>> 2010-02-16
>> 09:30:47
>>
>>
>>
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed
>> mode):
>>
>>
>>
>> "Attach Listener" daemon prio=10 tid=0x00007fbe9527f800 nid=0x4a48
>> waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State:
>> RUNNABLE
>>
>>
>>
>> "pool-1-thread-5" prio=10 tid=0x00007fbe95efd000 nid=0x6bab waiting on
>> condition [0x000000004238d000]
>>    java.lang.Thread.State: WAITING
>> (parking)
>>         at sun.misc.Unsafe.$$YJP$$park(Native
>> Method)
>>         - parking to wait for  <0x00007fbea529a0d8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>         at
>> sun.misc.Unsafe.park(Unsafe.java)
>>
>>
>>         at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>
>>
>>         at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
>>
>>
>>         at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>
>>
>>         at
>> java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB