Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - [winows] tracker file sharing voilation wne trying to delete tracker file


Copy link to this message
-
[winows] tracker file sharing voilation wne trying to delete tracker file
Phil Scala 2013-04-13, 00:07
Hi,

I am new to Flume, was trying out some basic's on my Windows 7 dev machine and I ran into problems with the spooling directory source, the issue was in the org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile() method... we were getting an instance of a DurablePositionTracker and not calling close on it before then creating another instance... thus not calling close on the underlying DataFileWriter.  This left the file locked in Windows and hence any delete attempts were failing...causing the Agent to spin around and around trying to delete this file and creating endless numbers of the temporary tracker files.

I have the code change here and can submit a JIRA item, I don't see anything similar in JIRA, but before I created something I wanted to discuss this in the community (realizing that Windows is not supported)

My code change is very simple, I added the close() call in the else block :

org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile()
....

if (!tracker.getTarget().equals(nextPath)) {
          tracker.close();
          deleteMetaFile();
        }
        else{
              //Explicitly close the tracker instance received so that
//there is no sharing violation on Windows.
              tracker.close();
        }
        tracker = DurablePositionTracker.getInstance(metaFile, nextPath);
....

Thanks
  Phil

Phil Scala
Developer / Architect
Global Relay

[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>

866.484.6630
New York  |  Chicago  |  Vancouver  |  London  (+44.0800.032.9829)  |  Singapore  (+65.3158.1301)
Global Relay Archive supports email, instant messaging, BlackBerry, Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, Facebook and more.

Ask about Global Relay Message<http://www.globalrelay.com/services/message> - The Future of Collaboration in the Financial Services World

All email sent to or from this address will be retained by Global Relay's email archiving system. This message is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law.  Global Relay will not be liable for any compliance or technical information provided herein.  All trademarks are the property of their respective owners.