Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> spoolDir source problem


+
Paul Chavez 2013-04-11, 22:15
+
Paul Chavez 2013-04-12, 18:41
+
Israel Ekpo 2013-04-12, 20:14
+
Paul Chavez 2013-04-12, 20:37
+
Israel Ekpo 2013-04-12, 20:42
+
Paul Chavez 2013-04-12, 20:48
Copy link to this message
-
Re: spoolDir source problem
Paul,
here is part of code which is throwing the exception
it is part of
flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java

    // On windows, things get messy with renames...
    // FIXME: This is not atomic. Consider implementing a recovery procedure
    // so that if it does not exist at startup, check for a rolled version
    // before creating a new file from scratch.
    if (PlatformDetect.isWindows()) {
      if (!trackerFile.delete()) {
        throw new IOException("Unable to delete existing meta file " +
            trackerFile);
      }
    }

I am not sure why the agent is not able to delete the file. Does the
agent have the permission to access those directories ? i mean both
read and write ?
I am no expert but just making a guess

On Sat, Apr 13, 2013 at 2:18 AM, Paul Chavez <
[EMAIL PROTECTED]> wrote:

> **
> We already have a CentOS cluster running half a dozen flume nodes, we've
> been feeding it production data for about 6 months and we've been very
> pleased with it so far. We are just looking to get agents on our app
> servers to smooth out cluster upgrades.
> Thanks for your help,
> Paul
>
>  ------------------------------
> *From:* Israel Ekpo [mailto:[EMAIL PROTECTED]]
> *Sent:* Friday, April 12, 2013 1:42 PM
>
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: spoolDir source problem
>
> It might be a good idea to set up Ubuntu 12 on a virtual machine using
> Virtual box and then set up your test environment there.
>
> This will give you some confidence that the set up works before you deploy
> it
>
> I dont really use Windows for development so unfortunately I am not able
> to help you troubleshoot this.
>
> On 12 April 2013 16:37, Paul Chavez <[EMAIL PROTECTED]>wrote:
>
>> **
>> 1. Flume 1.3.1 I believe, whatever is packaged with latest CDH
>> distribution.
>> 2. Windows Server 2008 R2
>> 3. The meta files are created by the flume agent, so should have full
>> rights. I'm went through and recreated the spool directory with more
>> explicit permissions now. It wasn't clear from the exception if the issue
>> was with the meta files or the files I'm putting in the spool dir.
>> Unfortunately it didn't seem to have an effect, recreated the directory
>> with full access for everyone and same issue.
>>
>> I'm ok with not having this functionality on Windows, just don't want to
>> waste time on a solution that won't work. My current solution uses the Avro
>> client to send files to a flume agent on our HDFS cluster running an avro
>> source. The main reason I want a local Windows agent is for the HTTP Source
>> which I've already been able to verify as working.
>>
>> Thanks,
>> Paul
>>
>>
>>  ------------------------------
>> *From:* Israel Ekpo [mailto:[EMAIL PROTECTED]]
>> *Sent:* Friday, April 12, 2013 1:15 PM
>> *To:* [EMAIL PROTECTED]
>> *Subject:* Re: spoolDir source problem
>>
>>   Paul,
>>
>> I have the following questions:
>>
>> (1) What version of Flume are you using?
>>
>> (2) What version of Windows are you using?
>>
>> (3) Does the user running Flume have permissions to read/write in the
>> directories used for the spooling and channels?
>>
>>
>> This will help narrow down the reasons why this could be happening.
>>
>> Nevertheless, it looks like the issue you are encountering is platform
>> specific (just on Windows)
>>
>> From your log messages, it appears the class in the calling thread is
>> org.apache.flume.client.avro.ReliableSpoolingFileEventReader
>>
>> However, the problem is happening in
>> org.apache.flume.serialization.DurablePositionTracker.getInstance()
>>
>> Within the source code, there is a comment on line 94 in the file stating
>> that on Windows renames is not really stable and the logic is not atomic.
>>
>> There is also a recommendation for implementing a recovery procedure so
>> that if the file does not exist on startup, it will check for a rolled
>> version before attempting to create a brand new file.
>>
Nitin Pawar
+
Paul Chavez 2013-04-12, 21:04
+
Paul Chavez 2013-04-12, 21:22
+
Paul Chavez 2013-04-12, 23:24
+
Israel Ekpo 2013-04-13, 01:15
+
Paul Chavez 2013-04-15, 18:37
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB