Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> FileSystem closed


Copy link to this message
-
Re: FileSystem closed
Do you close your FileSystem instances at all? IIRC, the FileSystem
instance you use is a singleton and if you close it once, it's closed
for everybody. My guess is you close it in your cleanup method and you
have JVM reuse turned on.

-Joey

On Thu, Sep 29, 2011 at 12:49 PM, Mark question <[EMAIL PROTECTED]> wrote:
> Hello,
>
>  I'm running 100 mappers sequentially on a single machine, where each
> mapper opens 100 files at the beginning then read one by one sequentially
> and closes after each one is done. After executing 6 mappers, the 7th gives
> this error:
>
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
>    at org.apache.hadoop.mapred.Child.main(Child.java:211)
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
>    at org.apache.hadoop.mapred.Child.main(Child.java:211)
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
>    at org.apache.hadoop.mapred.Child.main(Child.java:211)
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)

Joseph Echeverria
Cloudera, Inc.
443.305.9434