Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> .SpoolingFileLineReader warning....


Copy link to this message
-
.SpoolingFileLineReader warning....
First a bit of context, I'm using logrotate to monitor and copy (cp -p) log
files to a flume spooling directory source.  So every hour, logrotate
checks for and copies a file from the source to the flume destination. I
see the following warning message in the flume logs.
17 Nov 2012 14:47:07,682 WARN  [pool-10-thread-1]
(org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile:328)  -
Could not find file:
/mnt/flume/clickstream/clickstream.log-2012-11-17-1353163623
java.io.FileNotFoundException:
/mnt/flume/clickstream/clickstream.log-2012-11-17-1353163623 (Permission
denied)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileReader.<init>(FileReader.java:72)
at
org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFileLineReader.java:322)
at
org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:172)
at
org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Although it appears that Flume processes the log, I'm curious why I''m
seeing this and if I have anything with permissions incorrect?

Here's the permissions:

source log directory under /var/log:
drwxrwxr-x 2 ubuntu    ubuntu   4096 Nov 17 14:47 clickstream

source files:
-rw-rw-r-- 1 ubuntu ubuntu   9055750 Nov 17 13:29
clickstream.log-2012-11-17-1353158953.gz
-rw-rw-r-- 1 ubuntu ubuntu  13583565 Nov 17 14:17
clickstream.log-2012-11-17-1353161821.gz
-rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 14:47
clickstream.log-2012-11-17-1353163623
-rw-rw-r-- 1 ubuntu ubuntu  65648336 Nov 17 14:52 clickstream.log

flume source directory under /mnt/flume:
drwxrwxr-x 2 ubuntu ubuntu 4096 Nov 17 14:48 clickstream

flume source files:
-rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 13:29
clickstream.log-2012-11-17-1353158953.COMPLETED
-rw-rw-r-- 1 ubuntu ubuntu 196945008 Nov 17 14:17
clickstream.log-2012-11-17-1353161821.COMPLETED
-rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 14:47
clickstream.log-2012-11-17-1353163623.COMPLETED

Any insight would be appreciated.

Regards,

Dan
+
Brock Noland 2012-11-17, 15:57
+
Dan Young 2012-11-17, 16:01
+
Brock Noland 2012-11-17, 16:15
+
Dan Young 2012-11-17, 16:33
+
Dan Young 2012-11-19, 19:33
+
Patrick Wendell 2012-11-19, 23:03
+
Patrick Wendell 2012-11-19, 23:04
+
Brock Noland 2012-11-19, 23:29
+
Brock Noland 2012-11-20, 12:25
+
Dan Young 2012-11-20, 15:02
+
Brock Noland 2012-11-20, 16:21
+
Dan Young 2012-11-20, 16:59
+
Brock Noland 2012-11-20, 17:01
+
Dan Young 2012-11-20, 17:10
+
Brock Noland 2012-11-20, 17:14
+
Dan Young 2012-11-20, 17:17
+
Dan Young 2012-11-20, 20:03
+
Brock Noland 2012-11-20, 20:06
+
Patrick Wendell 2012-11-23, 12:46