Hello,

  I'm running 100 mappers sequentially on a single machine, where each
mapper opens 100 files at the beginning then read one by one sequentially
and closes after each one is done. After executing 6 mappers, the 7th gives
this error:

java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
    at java.io.FilterInputStream.close(FilterInputStream.java:155)
    at
org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
    at BFMapper20HM9.close(BFMapper20HM9.java:264)
    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
    at org.apache.hadoop.mapred.Child.main(Child.java:211)
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
    at java.io.FilterInputStream.close(FilterInputStream.java:155)
    at
org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
    at BFMapper20HM9.close(BFMapper20HM9.java:264)
    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
    at org.apache.hadoop.mapred.Child.main(Child.java:211)
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
    at java.io.FilterInputStream.close(FilterInputStream.java:155)
    at
org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
    at BFMapper20HM9.close(BFMapper20HM9.java:264)
    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
    at org.apache.hadoop.mapred.Child.main(Child.java:211)
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
    at java.io.FilterInputStream.close(FilterInputStream.java:155)
    at
org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
    at BFMapper20HM9.close(BFMapper20HM9.java:264)
    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
Can anybody give me a hint of what that could be?

Thank you,
Mark

Do you close your FileSystem instances at all? IIRC, the FileSystem
instance you use is a singleton and if you close it once, it's closed
for everybody. My guess is you close it in your cleanup method and you
have JVM reuse turned on.

-Joey

On Thu, Sep 29, 2011 at 12:49 PM, Mark question <[EMAIL PROTECTED]> wrote:
> Hello,
>
>  I'm running 100 mappers sequentially on a single machine, where each
> mapper opens 100 files at the beginning then read one by one sequentially
> and closes after each one is done. After executing 6 mappers, the 7th gives
> this error:
>
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
>    at org.apache.hadoop.mapred.Child.main(Child.java:211)
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
>    at org.apache.hadoop.mapred.Child.main(Child.java:211)
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
>    at org.apache.hadoop.mapred.Child.main(Child.java:211)
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)

Joseph Echeverria
Cloudera, Inc.
443.305.9434

FileSystem objects will be cached in jvm.
When it tries to get the FS object by using Filesystem.get(..) ( sequence file internally will use it), it will return same fs object if scheme and authority is same for the uri.

 fs cache key's equals implementation is below

 static boolean isEqual(Object a, Object b) {
        return a == b || (a != null && a.equals(b));        
      }

      /** {@inheritDoc} */
      public boolean equals(Object obj) {
        if (obj == this) {
          return true;
        }
        if (obj != null && obj instanceof Key) {
          Key that = (Key)obj;
          return isEqual(this.scheme, that.scheme)
                 && isEqual(this.authority, that.authority)
                 && isEqual(this.ugi, that.ugi)
                 && (this.unique == that.unique);
        }
        return false;        
      }
I think, here some your files uri and schems are same and got the same fs object. When it closes first one, diffenitely other will get this exception.

Regards,
Uma

----- Original Message -----
From: Joey Echeverria <[EMAIL PROTECTED]>
Date: Thursday, September 29, 2011 10:34 pm
Subject: Re: FileSystem closed
To: [EMAIL PROTECTED]

> Do you close your FileSystem instances at all? IIRC, the FileSystem
> instance you use is a singleton and if you close it once, it's closed
> for everybody. My guess is you close it in your cleanup method and you
> have JVM reuse turned on.
>
> -Joey
>
> On Thu, Sep 29, 2011 at 12:49 PM, Mark question
> <[EMAIL PROTECTED]> wrote:
> > Hello,
> >
> >  I'm running 100 mappers sequentially on a single machine, where
> each> mapper opens 100 files at the beginning then read one by one
> sequentially> and closes after each one is done. After executing 6
> mappers, the 7th gives
> > this error:
> >
> > java.io.IOException: Filesystem closed
> >    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
> >    at
> org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
> >    at
> >
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
> >    at BFMapper20HM9.close(BFMapper20HM9.java:264)
> >    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
> >    at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)>  
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> >    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
> >    at java.security.AccessController.doPrivileged(Native Method)
> >    at javax.security.auth.Subject.doAs(Subject.java:396)
> >    at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)>    at org.apache.hadoop.mapred.Child.main(Child.java:211)
> > java.io.IOException: Filesystem closed
> >    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
> >    at
> org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
> >    at
> >
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
> >    at BFMapper20HM9.close(BFMapper20HM9.java:264)
> >    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
> >    at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)>  
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> >    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
> >    at java.security.AccessController.doPrivileged(Native Method)
> >    at javax.security.auth.Subject.doAs(Subject.java:396)
> >    at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)>    at org.apache.hadoop.mapred.Child.main(Child.java:211)
> > java.io.IOException: Filesystem closed
> >    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
> >    at
> org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)>    at java.io.FilterInputStream.close(FilterInputStream.java:155)

On 29/09/2011 18:02, Joey Echeverria wrote:
> Do you close your FileSystem instances at all? IIRC, the FileSystem
> instance you use is a singleton and if you close it once, it's closed
> for everybody. My guess is you close it in your cleanup method and you
> have JVM reuse turned on.
I've hit this in the past. In 0.21+ you can ask for a new instance
explicity.

For 0.20.20x, set "fs.hdfs.impl.disable.cache" to true in the conf, and
new instances don't get cached.

Hello. I have a problem with the filesystem closing.

The filesystem was closed when the hive query is running.
It is 'select' query and the data size is about 1TB.
I'm using hadoop-0.20.2 and hive-0.7.1.

The error log is telling that tmp file is not deleted, or the tmp path
exception is occurred.

Is there any hadoop configuration I'm missing?

Thank you

[stderr logs]
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
Filesystem closed
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:454)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:636)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:557)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:226)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:617)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:453)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:648)
at org.apache.hadoop.fs.FileSystem.deleteOnExit(FileSystem.java:615)
at
org.apache.hadoop.hive.shims.Hadoop20Shims.fileSystemDeleteOnExit(Hadoop20Shims.java:68)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:451)
... 12 more