Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> hadoop FileSystem.close()


Copy link to this message
-
Re: hadoop FileSystem.close()
In all my experience you let FileSystem instances close themselves.

On Tue, Jul 24, 2012 at 10:34 AM, Koert Kuipers <[EMAIL PROTECTED]> wrote:
> Since FileSystem is a Closeable i would expect code using it to be like
> this:
>
> FileSystem fs = path.getFileSystem(conf);
> try {
>     // do something with fs, such as read from the path
> } finally {
>     fs.close()
> }
>
> However i have repeatedly gotten into trouble with this approach. In one
> situation it turned out that when i closed a FileSystem other operations
> that were using their own FileSystems (pointing to the same real-world HDFS
> filesystem) also saw their FileSystems closed, leading to very confusing
> read and write errors. This led me to believe that FileSystem should never
> be closed since it seemed to act like some sort of Singleton. However now
> was just looking at some code (Hoop server, to be precise) and noticed that
> FileSystems were indeed closed, but they were always threadlocal. Is this
> the right approach?
>
> And if FileSystem is threadlocal, is this safe (assuming fs1 and fs2 could
> point to the same real-world filesystem)?
>
> FileSystem fs1 = path.getFileSystem(conf);
> try {
>     FileSystem fs2 = path.getFileSystem(conf);
>     try {
>     // do something with fs2, such as read from the path
>     } finally {
>       fs2.close()
>     }
>     // do something with fs1, such as read from the path (note, fs2 is
> closed here, and i wouldn't be surprised if fs1 by now is also closed given
> my experience)
> } finally {
>   fs1.close()
> }
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB