Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> FileSystem closed


Copy link to this message
-
Re: FileSystem closed
Do you close your FileSystem instances at all? IIRC, the FileSystem
instance you use is a singleton and if you close it once, it's closed
for everybody. My guess is you close it in your cleanup method and you
have JVM reuse turned on.

-Joey

On Thu, Sep 29, 2011 at 12:49 PM, Mark question <[EMAIL PROTECTED]> wrote:
> Hello,
>
>  I'm running 100 mappers sequentially on a single machine, where each
> mapper opens 100 files at the beginning then read one by one sequentially
> and closes after each one is done. After executing 6 mappers, the 7th gives
> this error:
>
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
>    at org.apache.hadoop.mapred.Child.main(Child.java:211)
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
>    at org.apache.hadoop.mapred.Child.main(Child.java:211)
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
>    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>    at java.security.AccessController.doPrivileged(Native Method)
>    at javax.security.auth.Subject.doAs(Subject.java:396)
>    at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
>    at org.apache.hadoop.mapred.Child.main(Child.java:211)
> java.io.IOException: Filesystem closed
>    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
>    at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)
>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
>    at
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)
>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
>    at BFMapper20HM9.close(BFMapper20HM9.java:264)
>    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)

Joseph Echeverria
Cloudera, Inc.
443.305.9434
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB