Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> spoolDir source problem

Copy link to this message
Re: spoolDir source problem
here is part of code which is throwing the exception
it is part of

    // On windows, things get messy with renames...
    // FIXME: This is not atomic. Consider implementing a recovery procedure
    // so that if it does not exist at startup, check for a rolled version
    // before creating a new file from scratch.
    if (PlatformDetect.isWindows()) {
      if (!trackerFile.delete()) {
        throw new IOException("Unable to delete existing meta file " +

I am not sure why the agent is not able to delete the file. Does the
agent have the permission to access those directories ? i mean both
read and write ?
I am no expert but just making a guess

On Sat, Apr 13, 2013 at 2:18 AM, Paul Chavez <

> **
> We already have a CentOS cluster running half a dozen flume nodes, we've
> been feeding it production data for about 6 months and we've been very
> pleased with it so far. We are just looking to get agents on our app
> servers to smooth out cluster upgrades.
> Thanks for your help,
> Paul
>  ------------------------------
> *From:* Israel Ekpo [mailto:[EMAIL PROTECTED]]
> *Sent:* Friday, April 12, 2013 1:42 PM
> *Subject:* Re: spoolDir source problem
> It might be a good idea to set up Ubuntu 12 on a virtual machine using
> Virtual box and then set up your test environment there.
> This will give you some confidence that the set up works before you deploy
> it
> I dont really use Windows for development so unfortunately I am not able
> to help you troubleshoot this.
> On 12 April 2013 16:37, Paul Chavez <[EMAIL PROTECTED]>wrote:
>> **
>> 1. Flume 1.3.1 I believe, whatever is packaged with latest CDH
>> distribution.
>> 2. Windows Server 2008 R2
>> 3. The meta files are created by the flume agent, so should have full
>> rights. I'm went through and recreated the spool directory with more
>> explicit permissions now. It wasn't clear from the exception if the issue
>> was with the meta files or the files I'm putting in the spool dir.
>> Unfortunately it didn't seem to have an effect, recreated the directory
>> with full access for everyone and same issue.
>> I'm ok with not having this functionality on Windows, just don't want to
>> waste time on a solution that won't work. My current solution uses the Avro
>> client to send files to a flume agent on our HDFS cluster running an avro
>> source. The main reason I want a local Windows agent is for the HTTP Source
>> which I've already been able to verify as working.
>> Thanks,
>> Paul
>>  ------------------------------
>> *From:* Israel Ekpo [mailto:[EMAIL PROTECTED]]
>> *Sent:* Friday, April 12, 2013 1:15 PM
>> *Subject:* Re: spoolDir source problem
>>   Paul,
>> I have the following questions:
>> (1) What version of Flume are you using?
>> (2) What version of Windows are you using?
>> (3) Does the user running Flume have permissions to read/write in the
>> directories used for the spooling and channels?
>> This will help narrow down the reasons why this could be happening.
>> Nevertheless, it looks like the issue you are encountering is platform
>> specific (just on Windows)
>> From your log messages, it appears the class in the calling thread is
>> org.apache.flume.client.avro.ReliableSpoolingFileEventReader
>> However, the problem is happening in
>> org.apache.flume.serialization.DurablePositionTracker.getInstance()
>> Within the source code, there is a comment on line 94 in the file stating
>> that on Windows renames is not really stable and the logic is not atomic.
>> There is also a recommendation for implementing a recovery procedure so
>> that if the file does not exist on startup, it will check for a rolled
>> version before attempting to create a brand new file.
Nitin Pawar