Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Filesystem closed exception


Copy link to this message
-
Re: Filesystem closed exception
Alejandro Abdelnur 2013-01-30, 18:49
Hemanth,

Is FS caching enabled or not in your cluster?

A simple solution would be to modify your mapper code not to close the FS.
It will go away when the task ends anyway.

Thx
On Thu, Jan 24, 2013 at 5:26 PM, Hemanth Yamijala <[EMAIL PROTECTED]
> wrote:

> Hi,
>
> We are noticing a problem where we get a filesystem closed exception when
> a map task is done and is finishing execution. By map task, I literally
> mean the MapTask class of the map reduce code. Debugging this we found that
> the mapper is getting a handle to the filesystem object and itself calling
> a close on it. Because filesystem objects are cached, I believe the
> behaviour is as expected in terms of the exception.
>
> I just wanted to confirm that:
>
> - if we do have a requirement to use a filesystem object in a mapper or
> reducer, we should either not close it ourselves
> - or (seems better to me) ask for a new version of the filesystem instance
> by setting the fs.hdfs.impl.disable.cache property to true in job
> configuration.
>
> Also, does anyone know if this behaviour was any different in Hadoop 0.20 ?
>
> For some context, this behaviour is actually seen in Oozie, which runs a
> launcher mapper for a simple java action. Hence, the java action could very
> well interact with a file system. I know this is probably better addressed
> in Oozie context, but wanted to get the map reduce view of things.
>
>
> Thanks,
> Hemanth
>

--
Alejandro