Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # dev >> [jira] [Commented] (AVRO-1144) Deadlock with FSInput and Hadoop NativeS3FileSystem.


Copy link to this message
-
[jira] [Commented] (AVRO-1144) Deadlock with FSInput and Hadoop NativeS3FileSystem.

    [ https://issues.apache.org/jira/browse/AVRO-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881022#comment-13881022 ]

Zhenxiao Luo commented on AVRO-1144:
------------------------------------

Hi Scott and Shawn,

I found exactly the same problem when running Presto(an open source SQL engine built on top of Hadoop) against NativeS3FileSystem. Could not get it fixed now. Currently I am using jets3t 0.6, which is using httpclient 3.1.

Just wonder, would update to jets3t 0.9 get the problem fixed?

Thanks,
Zhenxiao

> Deadlock with FSInput and Hadoop NativeS3FileSystem.
> ----------------------------------------------------
>
>                 Key: AVRO-1144
>                 URL: https://issues.apache.org/jira/browse/AVRO-1144
>             Project: Avro
>          Issue Type: Bug
>          Components: java
>    Affects Versions: 1.7.0
>         Environment: Hadoop 1.0.3
>            Reporter: Shawn Smith
>            Assignee: Scott Carey
>             Fix For: 1.7.5
>
>         Attachments: AVRO-1144.patch
>
>
> Deadlock can occur when using org.apache.avro.mapred.FsInput to read files from S3 using the Hadoop NativeS3FileSystem and multiple threads.
> There are a lot of components involved, but the basic cause is pretty simple: Apache Commons HttpClient can deadlock waiting for a free HTTP connection when the number of threads downloading from S3 is greater than or equal to the maximum allowed HTTP connections per host.
> I've filed this bug against Avro because the bug is easiest to fix in Avro.  Swap the order of the FileSystem.open() and FileSystem.getFileStatus() calls in the FSInput constructor:
> {noformat}
> /** Construct given a path and a configuration. */
> public FsInput(Path path, Configuration conf) throws IOException {
>   this.stream = path.getFileSystem(conf).open(path);
>   this.len = path.getFileSystem(conf).getFileStatus(path).getLen();
> }
> {noformat}
> to
> {noformat}
> /** Construct given a path and a configuration. */
> public FsInput(Path path, Configuration conf) throws IOException {
>   this.len = path.getFileSystem(conf).getFileStatus(path).getLen();
>   this.stream = path.getFileSystem(conf).open(path);
> }
> {noformat}
> Here's what triggers the deadlock:
> * FSInput calls FileSystem.open() which calls Jets3t to connect to S3 and open an HTTP connection for downloading content.  This acquires an HTTP connection but does not release it.
> * FSInput calls FileSystem.getFileStatus() which calls Jets3t to connect to S3 and perform a HEAD request to get object metadata.  This attempts to acquire a second HTTP connection.
> * Jets3t uses Apache Commons HTTP Client which limits the number of simultaneous HTTP connections to a given host.  Lets say this maximum is 4 (the default)...  If 4 threads all call the FSInput constructor concurrently, the 4 FileSystem.open() calls can acquire all 4 available connections and the FileSystem.getFileStatus() calls block forever waiting for a thread to release an HTTP connection back to the connection pool.
> A simple way to reproduce the problem this problem is to create "jets3t.properties" in your classpath with "httpclient.max-connections=1".  Then try to open a file using FSInput and the Native S3 file system (new Path("s3n://<bucket>/<path>")).  It will hang indefinitely inside the FSInput constructor.
> Swapping the order of the open() and getFileStatus() calls ensures that a given thread using FSInput has at most one outstanding connection S3 at a time.  As a result, one thread should always be able to make progress, avoiding deadlock.
> Here's a sample stack trace of a deadlocked thread:
> {noformat}
> "pool-10-thread-3" prio=5 tid=11026f800 nid=0x116a04000 in Object.wait() [116a02000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <785892cc0> (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
> at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518)

This message was sent by Atlassian JIRA
(v6.1.5#6160)

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB