Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - .SpoolingFileLineReader warning....


+
Dan Young 2012-11-17, 15:02
+
Brock Noland 2012-11-17, 15:57
+
Dan Young 2012-11-17, 16:01
+
Brock Noland 2012-11-17, 16:15
+
Dan Young 2012-11-17, 16:33
Copy link to this message
-
Re: .SpoolingFileLineReader warning....
Dan Young 2012-11-19, 19:33
Hello Brock,

It seems like we get this message each time that logrotate runs and is in
the process of copying the file to the SpoolingDirectory. It seems that
Flume starts reading the file as soon as it shows up in the
SpoolingDirectory.....  Maybe it's trying to read the file while it's still
being written to????

2012-11-19 19:27:27,924 (pool-12-thread-1) [WARN -
org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFileLineReader.java:328)]
Could not find file:
/mnt/flume/clickstream2/clickstream2.log-2012-11-19-1353353239
java.io.FileNotFoundException:
/mnt/flume/clickstream2/clickstream2.log-2012-11-19-1353353239 (Permission
denied)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileReader.<init>(FileReader.java:72)
at
org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFileLineReader.java:322)
at
org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:172)
at
org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
On Sat, Nov 17, 2012 at 9:15 AM, Brock Noland <[EMAIL PROTECTED]> wrote:

>  Ok, do you mind sharing your log rotate config to see if we can
> reproduce?
>
> --
> Brock Noland
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> On Saturday, November 17, 2012 at 10:01 AM, Dan Young wrote:
>
> Hey Brock,
>
> No I have not modified the conf while the agent was running.
>
> /mnt/flume is local. Note that this is running on an ec2 instance and the
> disk is the ephemeral drive, not EBS.
>
> Regards ,
>
> Dano
> On Nov 17, 2012 8:58 AM, "Brock Noland" <[EMAIL PROTECTED]> wrote:
>
> Hi,
>
> I highly doubt it's related to
> (https://issues.apache.org/jira/browse/FLUME-1721) but have you
> modified the configuration file since starting the agent?  If so, can
> you restart the agent and see if the error continues?
>
> Also, is /mnt/flume local disk or NAS?
>
> Brock
>
> On Sat, Nov 17, 2012 at 9:02 AM, Dan Young <[EMAIL PROTECTED]> wrote:
> > First a bit of context, I'm using logrotate to monitor and copy (cp -p)
> log
> > files to a flume spooling directory source.  So every hour, logrotate
> checks
> > for and copies a file from the source to the flume destination. I see the
> > following warning message in the flume logs.
> >
> >
> > 17 Nov 2012 14:47:07,682 WARN  [pool-10-thread-1]
> > (org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile:328)  -
> > Could not find file:
> > /mnt/flume/clickstream/clickstream.log-2012-11-17-1353163623
> > java.io.FileNotFoundException:
> > /mnt/flume/clickstream/clickstream.log-2012-11-17-1353163623 (Permission
> > denied)
> > at java.io.FileInputStream.open(Native Method)
> > at java.io.FileInputStream.<init>(FileInputStream.java:138)
> > at java.io.FileReader.<init>(FileReader.java:72)
> > at
> >
> org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFileLineReader.java:322)
> > at
> >
> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:172)
> > at
> >
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
+
Patrick Wendell 2012-11-19, 23:03
+
Patrick Wendell 2012-11-19, 23:04
+
Brock Noland 2012-11-19, 23:29
+
Brock Noland 2012-11-20, 12:25
+
Dan Young 2012-11-20, 15:02
+
Brock Noland 2012-11-20, 16:21
+
Dan Young 2012-11-20, 16:59
+
Brock Noland 2012-11-20, 17:01
+
Dan Young 2012-11-20, 17:10
+
Brock Noland 2012-11-20, 17:14
+
Dan Young 2012-11-20, 17:17
+
Dan Young 2012-11-20, 20:03
+
Brock Noland 2012-11-20, 20:06
+
Patrick Wendell 2012-11-23, 12:46