Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> spoolDir source problem


+
Paul Chavez 2013-04-11, 22:15
+
Paul Chavez 2013-04-12, 18:41
+
Israel Ekpo 2013-04-12, 20:14
+
Paul Chavez 2013-04-12, 20:37
+
Israel Ekpo 2013-04-12, 20:42
+
Paul Chavez 2013-04-12, 20:48
Copy link to this message
-
Re: spoolDir source problem
Paul,
here is part of code which is throwing the exception
it is part of
flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java

    // On windows, things get messy with renames...
    // FIXME: This is not atomic. Consider implementing a recovery procedure
    // so that if it does not exist at startup, check for a rolled version
    // before creating a new file from scratch.
    if (PlatformDetect.isWindows()) {
      if (!trackerFile.delete()) {
        throw new IOException("Unable to delete existing meta file " +
            trackerFile);
      }
    }

I am not sure why the agent is not able to delete the file. Does the
agent have the permission to access those directories ? i mean both
read and write ?
I am no expert but just making a guess

On Sat, Apr 13, 2013 at 2:18 AM, Paul Chavez <
[EMAIL PROTECTED]> wrote:

> **
> We already have a CentOS cluster running half a dozen flume nodes, we've
> been feeding it production data for about 6 months and we've been very
> pleased with it so far. We are just looking to get agents on our app
> servers to smooth out cluster upgrades.
> Thanks for your help,
> Paul
>
>  ------------------------------
> *From:* Israel Ekpo [mailto:[EMAIL PROTECTED]]
> *Sent:* Friday, April 12, 2013 1:42 PM
>
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: spoolDir source problem
>
> It might be a good idea to set up Ubuntu 12 on a virtual machine using
> Virtual box and then set up your test environment there.
>
> This will give you some confidence that the set up works before you deploy
> it
>
> I dont really use Windows for development so unfortunately I am not able
> to help you troubleshoot this.
>
> On 12 April 2013 16:37, Paul Chavez <[EMAIL PROTECTED]>wrote:
>
>> **
>> 1. Flume 1.3.1 I believe, whatever is packaged with latest CDH
>> distribution.
>> 2. Windows Server 2008 R2
>> 3. The meta files are created by the flume agent, so should have full
>> rights. I'm went through and recreated the spool directory with more
>> explicit permissions now. It wasn't clear from the exception if the issue
>> was with the meta files or the files I'm putting in the spool dir.
>> Unfortunately it didn't seem to have an effect, recreated the directory
>> with full access for everyone and same issue.
>>
>> I'm ok with not having this functionality on Windows, just don't want to
>> waste time on a solution that won't work. My current solution uses the Avro
>> client to send files to a flume agent on our HDFS cluster running an avro
>> source. The main reason I want a local Windows agent is for the HTTP Source
>> which I've already been able to verify as working.
>>
>> Thanks,
>> Paul
>>
>>
>>  ------------------------------
>> *From:* Israel Ekpo [mailto:[EMAIL PROTECTED]]
>> *Sent:* Friday, April 12, 2013 1:15 PM
>> *To:* [EMAIL PROTECTED]
>> *Subject:* Re: spoolDir source problem
>>
>>   Paul,
>>
>> I have the following questions:
>>
>> (1) What version of Flume are you using?
>>
>> (2) What version of Windows are you using?
>>
>> (3) Does the user running Flume have permissions to read/write in the
>> directories used for the spooling and channels?
>>
>>
>> This will help narrow down the reasons why this could be happening.
>>
>> Nevertheless, it looks like the issue you are encountering is platform
>> specific (just on Windows)
>>
>> From your log messages, it appears the class in the calling thread is
>> org.apache.flume.client.avro.ReliableSpoolingFileEventReader
>>
>> However, the problem is happening in
>> org.apache.flume.serialization.DurablePositionTracker.getInstance()
>>
>> Within the source code, there is a comment on line 94 in the file stating
>> that on Windows renames is not really stable and the logic is not atomic.
>>
>> There is also a recommendation for implementing a recovery procedure so
>> that if the file does not exist on startup, it will check for a rolled
>> version before attempting to create a brand new file.
>>
Nitin Pawar
+
Paul Chavez 2013-04-12, 21:04
+
Paul Chavez 2013-04-12, 21:22
+
Paul Chavez 2013-04-12, 23:24
+
Israel Ekpo 2013-04-13, 01:15
+
Paul Chavez 2013-04-15, 18:37