Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> spoolDir source problem


Copy link to this message
-
Re: spoolDir source problem
Paul,

I have the following questions:

(1) What version of Flume are you using?

(2) What version of Windows are you using?

(3) Does the user running Flume have permissions to read/write in the
directories used for the spooling and channels?
This will help narrow down the reasons why this could be happening.

Nevertheless, it looks like the issue you are encountering is platform
specific (just on Windows)

>From your log messages, it appears the class in the calling thread is
org.apache.flume.client.avro.ReliableSpoolingFileEventReader

However, the problem is happening in
org.apache.flume.serialization.DurablePositionTracker.getInstance()

Within the source code, there is a comment on line 94 in the file stating
that on Windows renames is not really stable and the logic is not atomic.

There is also a recommendation for implementing a recovery procedure so
that if the file does not exist on startup, it will check for a rolled
version before attempting to create a brand new file.

If it is possible for you to move to a different environment other than
Windows, that would be great.

If this is not possible, then try deleting your spooling directory
"c:\flume_data\spool\web" which will also remove the metadata files
recursively.

Back up all the pending files that have not yet been processed in the
spooling directory before deleting the folder so that you can put the files
back after the directory is recreated.

Then restart your agent to see if this works.

Let me know if this helps.

On 12 April 2013 14:41, Paul Chavez <[EMAIL PROTECTED]> wrote:

> **
> Anyone have any ideas on this? I can't even find the class throwing the
> exception to try and see what it is doing. I would really like to use this
> on Windows, but would like to know at least if there's some compatibility
> issue so I can move on.
>
> thanks,
> Paul
>
>
>  ------------------------------
> *From:* Paul Chavez [mailto:[EMAIL PROTECTED]]
> *Sent:* Thursday, April 11, 2013 3:15 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* spoolDir source problem
>
>  Hello,
>
> I've run into a problem with the spoolDir source, on Windows, and am not
> sure how to proceed.
>
> The agent starts fine and the source is created without issue and is
> apparently ready. After agent start a .flumespool directory is created in
> the path the source is watching. This directory remains empty as long as
> the agent is idle.
>
> However, as soon as I drop a file into the spool directory (parent to the
> .flumespool directory), I get a series of errors in the flume log and a
> file named '.flumespool-main.meta<string of numbers>.tmp' is created in
> that .flumespool directory at the rate of one per second. The file in the
> spool directory is never touched as far as I can tell and the /metrics web
> page shows no movement on the channel or sink. A possibly related note is
> that the sources don't show in the metrics page, even though the logs say
> the source(s) are started.
>
> All I have done so far is set the directory security to be 'Everyone/Full
> Control', basically the windows version of 'chmod 777'
>
> Any help is appreciated!
>
> thanks,
> Paul
>
> Here's what the log shows.
> 11 Apr 2013 15:11:48,092 INFO  [conf-file-poller-0]
> (org.apache.flume.node.Application.startAllComponents:184)  - Starting
> Source spool_WebLogs
> 11 Apr 2013 15:11:48,092 INFO  [conf-file-poller-0]
> (org.apache.flume.node.Application.startAllComponents:184)  - Starting
> Source http_Default
> 11 Apr 2013 15:11:48,092 INFO  [lifecycleSupervisor-1-0]
> (org.apache.flume.source.SpoolDirectorySource.start:66)  -
> SpoolDirectorySource source starting with directory: c:\flume_data\spool\web
> 11 Apr 2013 15:11:48,124 INFO  [conf-file-poller-0] (
> org.mortbay.log.Slf4jLog.info:67)  - Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 11 Apr 2013 15:11:48,139 INFO  [conf-file-poller-0] (
> org.mortbay.log.Slf4jLog.info:67)  - jetty-6.1.26