Historically, there has never been a firm requirement that clients must call FileSystem#close upon finishing usage of an instance. I think the history here is that the close method was not part of the initial API definition, and when it was added, there were already a lot of existing applications coded that were not calling the close method. A tactical decision was made not to force all of those applications to make code changes, so calling close has been treated as optional. (This all pre-dates my time on the project though, so I might not have the story completely right.)
I'd prefer a hard requirement that applications must call close, but this is our current situation. If you see a specific point in the codebase where you'd like to propose adding a close call, then I recommend filing an Apache JIRA to discuss the proposal. If you'd like, also consider attaching a patch to the JIRA.
From: M G <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Wednesday, July 13, 2016 at 7:23 AM
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Fwd: FileSystem object's close method is not called. hadoop 2.7.2
Hi, I have implemented an HC file system by extending the org.apache.hadoop.fs.FileSystem class.
Things seem to work fine, except the fact that when I run mapreduce jobs with yarn (simple DFSIO tests) I keep noticing that my Filesystem's "close" method is not called.
That is, I see in logs that my file system object is instantiated seven times during the run of the MRAppMaster process, but the close method was never closed for any instantiation.
I know that there is a configuration key fs.automatic.close that is checked on MRappMaster.java - and I make sure that this key is true.
I am using hadoop 2.7.2
Other processes that instantiate my FS class (node manager for example) are calling the close method, I saw the problem only with MRAppMaster's process.
The DFSIO test succeeds, but this issue can lead to some resources leakage so it's important that the close method is called.