Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - RE: Is FileSystem thread-safe?


Copy link to this message
-
Re: Is FileSystem thread-safe?
Matthew Farrellee 2013-04-02, 11:41
If you're interested in the semantics of FileSystem operations, have a
look a HADOOP-9371[0]

Depending on what you're trying to do, the thread-safety of a particular
FS implementation in a single JVM instance may not be as important as
the semantics you get across JVM instances.

Best,
matt

[0] https://issues.apache.org/jira/browse/HADOOP-9371
On 04/01/2013 06:17 PM, Arpit Agarwal wrote:
> Hi John,
>
> DistributedFileSystem is intended to be thread-safe, true to its name.
>
> Metadata operations are handled by the NameNode server which
> synchronizes concurrent client requests via locks (you can look at the
> FSNameSystem class).
>
> Some discussion on the thread-safety aspects of HDFS:
> http://storageconference.org/2010/Papers/MSST/Shvachko.pdf
>
> -Arpit
>
>
> On Sun, Mar 31, 2013 at 11:52 AM, Ted Yu <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     If you look at DistributedFileSystem source code, you would see that
>     it calls the DFSClient field member for most of the actions.
>     Requests to Namenode are then made through ClientProtocol.
>
>     An hdfs committer would be able to give you affirmative answer.
>
>
>     On Sun, Mar 31, 2013 at 11:27 AM, John Lilley
>     <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
>
>         *From:*Ted Yu [mailto:[EMAIL PROTECTED]
>         <mailto:[EMAIL PROTECTED]>]
>         *Subject:* Re: Is FileSystem thread-safe?____
>
>         >>FileSystem is an abstract class, what concrete class are you
>         using (DistributedFileSystem, etc) ? ____
>
>         Good point.  I am calling FileSystem.get(URI uri, Configuration
>         conf) with an URI like �hdfs://server:port/�� on a remote
>         server, so I assume it is creating a DistributedFileSystem.
>         However I am not finding any documentation discussing its
>         thread-safety (or lack thereof), perhaps you can point me to it?____
>
>         Thanks, john____
>
>
>