Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> upload hang at DFSClient$DFSOutputStream.close(3488)


Copy link to this message
-
RE: upload hang at DFSClient$DFSOutputStream.close(3488)
Hi Mingxi,

In your thread dump, did you check DataStreamer thread? is it running?

If DataStreamer thread is not running, then this issue would be mostly same as HDFS-2850.

Did you find any OOME in your clients?

Regards,
Uma
________________________________________
From: Mingxi Wu [[EMAIL PROTECTED]]
Sent: Monday, April 16, 2012 7:25 AM
To: [EMAIL PROTECTED]
Subject: upload hang at DFSClient$DFSOutputStream.close(3488)

Hi,

I use hadoop cloudera 0.20.2-cdh3u0.

I have a program which uploads local files to HDFS every hour.

Basically, I open a gzip input stream by in= new GZIPInputStream(fin); And write to HDFS file. After less than two days, it will hang. It hangs at FSDataOutputStream.close(86).
Here is the stack:

State: WAITING Running 16660 ms (user 13770 ms) blocked 11276 times for <> ms waiting 11209 times for <> ms
LockName: java.util.LinkedList@f1ca0de LockOwnerId: -1
java.lang.Object.wait(-2)
java.lang.Object.wait(485)
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(3468)
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal(3457)
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(3549)
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(3488)
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(61)
org.apache.hadoop.fs.FSDataOutputStream.close(86)
org.apache.hadoop.io.IOUtils.copyBytes(59)
org.apache.hadoop.io.IOUtils.copyBytes(74)

Any suggestion to avoid this issue? It seems this is a bug in hadoop. I found this issue is less severe when my upload server do one upload at a time, instead of using multiple concurrent uploads.

Thanks,

Mingxi