Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> RE: Is FileSystem thread-safe?


Copy link to this message
-
Re: Is FileSystem thread-safe?
If you're interested in the semantics of FileSystem operations, have a
look a HADOOP-9371[0]

Depending on what you're trying to do, the thread-safety of a particular
FS implementation in a single JVM instance may not be as important as
the semantics you get across JVM instances.

Best,
matt

[0] https://issues.apache.org/jira/browse/HADOOP-9371
On 04/01/2013 06:17 PM, Arpit Agarwal wrote:
> Hi John,
>
> DistributedFileSystem is intended to be thread-safe, true to its name.
>
> Metadata operations are handled by the NameNode server which
> synchronizes concurrent client requests via locks (you can look at the
> FSNameSystem class).
>
> Some discussion on the thread-safety aspects of HDFS:
> http://storageconference.org/2010/Papers/MSST/Shvachko.pdf
>
> -Arpit
>
>
> On Sun, Mar 31, 2013 at 11:52 AM, Ted Yu <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     If you look at DistributedFileSystem source code, you would see that
>     it calls the DFSClient field member for most of the actions.
>     Requests to Namenode are then made through ClientProtocol.
>
>     An hdfs committer would be able to give you affirmative answer.
>
>
>     On Sun, Mar 31, 2013 at 11:27 AM, John Lilley
>     <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
>
>         *From:*Ted Yu [mailto:[EMAIL PROTECTED]
>         <mailto:[EMAIL PROTECTED]>]
>         *Subject:* Re: Is FileSystem thread-safe?____
>
>         >>FileSystem is an abstract class, what concrete class are you
>         using (DistributedFileSystem, etc) ? ____
>
>         Good point.  I am calling FileSystem.get(URI uri, Configuration
>         conf) with an URI like �hdfs://server:port/�� on a remote
>         server, so I assume it is creating a DistributedFileSystem.
>         However I am not finding any documentation discussing its
>         thread-safety (or lack thereof), perhaps you can point me to it?____
>
>         Thanks, john____
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB