Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> spoolDir source problem


Copy link to this message
-
RE: spoolDir source problem
Anyone have any ideas on this? I can't even find the class throwing the exception to try and see what it is doing. I would really like to use this on Windows, but would like to know at least if there's some compatibility issue so I can move on.

thanks,
Paul
________________________________
From: Paul Chavez [mailto:[EMAIL PROTECTED]]
Sent: Thursday, April 11, 2013 3:15 PM
To: [EMAIL PROTECTED]
Subject: spoolDir source problem

Hello,

I've run into a problem with the spoolDir source, on Windows, and am not sure how to proceed.

The agent starts fine and the source is created without issue and is apparently ready. After agent start a .flumespool directory is created in the path the source is watching. This directory remains empty as long as the agent is idle.

However, as soon as I drop a file into the spool directory (parent to the .flumespool directory), I get a series of errors in the flume log and a file named '.flumespool-main.meta<string of numbers>.tmp' is created in that .flumespool directory at the rate of one per second. The file in the spool directory is never touched as far as I can tell and the /metrics web page shows no movement on the channel or sink. A possibly related note is that the sources don't show in the metrics page, even though the logs say the source(s) are started.

All I have done so far is set the directory security to be 'Everyone/Full Control', basically the windows version of 'chmod 777'

Any help is appreciated!

thanks,
Paul

Here's what the log shows.
11 Apr 2013 15:11:48,092 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184)  - Starting Source spool_WebLogs
11 Apr 2013 15:11:48,092 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184)  - Starting Source http_Default
11 Apr 2013 15:11:48,092 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.source.SpoolDirectorySource.start:66)  - SpoolDirectorySource source starting with directory: c:\flume_data\spool\web
11 Apr 2013 15:11:48,124 INFO  [conf-file-poller-0] (org.mortbay.log.Slf4jLog.info:67)  - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
11 Apr 2013 15:11:48,139 INFO  [conf-file-poller-0] (org.mortbay.log.Slf4jLog.info:67)  - jetty-6.1.26
11 Apr 2013 15:11:48,155 INFO  [conf-file-poller-0] (org.mortbay.log.Slf4jLog.info:67)  - Started SocketConnector@0.0.0.0:41414<mailto:SocketConnector@0.0.0.0:41414>
11 Apr 2013 15:11:48,202 INFO  [lifecycleSupervisor-1-2] (org.mortbay.log.Slf4jLog.info:67)  - jetty-6.1.26
11 Apr 2013 15:11:48,217 INFO  [lifecycleSupervisor-1-2] (org.mortbay.log.Slf4jLog.info:67)  - Started SocketConnector@0.0.0.0:6240<mailto:SocketConnector@0.0.0.0:6240>
11 Apr 2013 15:11:48,389 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.sink.AvroSink.start:253)  - Avro sink avro_Default started.
11 Apr 2013 15:11:48,404 ERROR [pool-6-thread-1] (org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile:442)  - Exception opening file: c:\flume_data\spool\web\u_ex130411.log-201304111500.log
java.io.IOException: Unable to delete existing meta file c:\flume_data\spool\web\.flumespool\.flumespool-main.meta
 at org.apache.flume.serialization.DurablePositionTracker.getInstance(DurablePositionTracker.java:96)
 at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile(ReliableSpoolingFileEventReader.java:423)
 at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:212)
 at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:154)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
 at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
11 Apr 2013 15:11:48,919 ERROR [pool-6-thread-1] (org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile:442)  - Exception opening file: c:\flume_data\spool\web\u_ex130411.log-201304111500.log
java.io.IOException: Unable to delete existing meta file c:\flume_data\spool\web\.flumespool\.flumespool-main.meta
 at org.apache.flume.serialization.DurablePositionTracker.getInstance(DurablePositionTracker.java:96)
 at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile(ReliableSpoolingFileEventReader.java:417)
 at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:212)
 at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:154)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
 at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB