Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # dev >> [jira] [Commented] (AVRO-1144) Deadlock with FSInput and Hadoop NativeS3FileSystem.


Copy link to this message
-
[jira] [Commented] (AVRO-1144) Deadlock with FSInput and Hadoop NativeS3FileSystem.

    [ https://issues.apache.org/jira/browse/AVRO-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881022#comment-13881022 ]

Zhenxiao Luo commented on AVRO-1144:
------------------------------------

Hi Scott and Shawn,

I found exactly the same problem when running Presto(an open source SQL engine built on top of Hadoop) against NativeS3FileSystem. Could not get it fixed now. Currently I am using jets3t 0.6, which is using httpclient 3.1.

Just wonder, would update to jets3t 0.9 get the problem fixed?

Thanks,
Zhenxiao

> Deadlock with FSInput and Hadoop NativeS3FileSystem.
> ----------------------------------------------------
>
>                 Key: AVRO-1144
>                 URL: https://issues.apache.org/jira/browse/AVRO-1144
>             Project: Avro
>          Issue Type: Bug
>          Components: java
>    Affects Versions: 1.7.0
>         Environment: Hadoop 1.0.3
>            Reporter: Shawn Smith
>            Assignee: Scott Carey
>             Fix For: 1.7.5
>
>         Attachments: AVRO-1144.patch
>
>
> Deadlock can occur when using org.apache.avro.mapred.FsInput to read files from S3 using the Hadoop NativeS3FileSystem and multiple threads.
> There are a lot of components involved, but the basic cause is pretty simple: Apache Commons HttpClient can deadlock waiting for a free HTTP connection when the number of threads downloading from S3 is greater than or equal to the maximum allowed HTTP connections per host.
> I've filed this bug against Avro because the bug is easiest to fix in Avro.  Swap the order of the FileSystem.open() and FileSystem.getFileStatus() calls in the FSInput constructor:
> {noformat}
> /** Construct given a path and a configuration. */
> public FsInput(Path path, Configuration conf) throws IOException {
>   this.stream = path.getFileSystem(conf).open(path);
>   this.len = path.getFileSystem(conf).getFileStatus(path).getLen();
> }
> {noformat}
> to
> {noformat}
> /** Construct given a path and a configuration. */
> public FsInput(Path path, Configuration conf) throws IOException {
>   this.len = path.getFileSystem(conf).getFileStatus(path).getLen();
>   this.stream = path.getFileSystem(conf).open(path);
> }
> {noformat}
> Here's what triggers the deadlock:
> * FSInput calls FileSystem.open() which calls Jets3t to connect to S3 and open an HTTP connection for downloading content.  This acquires an HTTP connection but does not release it.
> * FSInput calls FileSystem.getFileStatus() which calls Jets3t to connect to S3 and perform a HEAD request to get object metadata.  This attempts to acquire a second HTTP connection.
> * Jets3t uses Apache Commons HTTP Client which limits the number of simultaneous HTTP connections to a given host.  Lets say this maximum is 4 (the default)...  If 4 threads all call the FSInput constructor concurrently, the 4 FileSystem.open() calls can acquire all 4 available connections and the FileSystem.getFileStatus() calls block forever waiting for a thread to release an HTTP connection back to the connection pool.
> A simple way to reproduce the problem this problem is to create "jets3t.properties" in your classpath with "httpclient.max-connections=1".  Then try to open a file using FSInput and the Native S3 file system (new Path("s3n://<bucket>/<path>")).  It will hang indefinitely inside the FSInput constructor.
> Swapping the order of the open() and getFileStatus() calls ensures that a given thread using FSInput has at most one outstanding connection S3 at a time.  As a result, one thread should always be able to make progress, avoiding deadlock.
> Here's a sample stack trace of a deadlocked thread:
> {noformat}
> "pool-10-thread-3" prio=5 tid=11026f800 nid=0x116a04000 in Object.wait() [116a02000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <785892cc0> (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
> at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518)

This message was sent by Atlassian JIRA
(v6.1.5#6160)