Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> .SpoolingFileLineReader warning....


+
Dan Young 2012-11-17, 15:02
Copy link to this message
-
Re: .SpoolingFileLineReader warning....
Hi,

I highly doubt it's related to
(https://issues.apache.org/jira/browse/FLUME-1721) but have you
modified the configuration file since starting the agent?  If so, can
you restart the agent and see if the error continues?

Also, is /mnt/flume local disk or NAS?

Brock

On Sat, Nov 17, 2012 at 9:02 AM, Dan Young <[EMAIL PROTECTED]> wrote:
> First a bit of context, I'm using logrotate to monitor and copy (cp -p) log
> files to a flume spooling directory source.  So every hour, logrotate checks
> for and copies a file from the source to the flume destination. I see the
> following warning message in the flume logs.
>
>
> 17 Nov 2012 14:47:07,682 WARN  [pool-10-thread-1]
> (org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile:328)  -
> Could not find file:
> /mnt/flume/clickstream/clickstream.log-2012-11-17-1353163623
> java.io.FileNotFoundException:
> /mnt/flume/clickstream/clickstream.log-2012-11-17-1353163623 (Permission
> denied)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at java.io.FileReader.<init>(FileReader.java:72)
> at
> org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFileLineReader.java:322)
> at
> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:172)
> at
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
>
>
> Although it appears that Flume processes the log, I'm curious why I''m
> seeing this and if I have anything with permissions incorrect?
>
>
>
> Here's the permissions:
>
> source log directory under /var/log:
> drwxrwxr-x 2 ubuntu    ubuntu   4096 Nov 17 14:47 clickstream
>
> source files:
> -rw-rw-r-- 1 ubuntu ubuntu   9055750 Nov 17 13:29
> clickstream.log-2012-11-17-1353158953.gz
> -rw-rw-r-- 1 ubuntu ubuntu  13583565 Nov 17 14:17
> clickstream.log-2012-11-17-1353161821.gz
> -rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 14:47
> clickstream.log-2012-11-17-1353163623
> -rw-rw-r-- 1 ubuntu ubuntu  65648336 Nov 17 14:52 clickstream.log
>
> flume source directory under /mnt/flume:
> drwxrwxr-x 2 ubuntu ubuntu 4096 Nov 17 14:48 clickstream
>
> flume source files:
> -rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 13:29
> clickstream.log-2012-11-17-1353158953.COMPLETED
> -rw-rw-r-- 1 ubuntu ubuntu 196945008 Nov 17 14:17
> clickstream.log-2012-11-17-1353161821.COMPLETED
> -rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 14:47
> clickstream.log-2012-11-17-1353163623.COMPLETED
>
> Any insight would be appreciated.
>
> Regards,
>
> Dan

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
+
Dan Young 2012-11-17, 16:01
+
Brock Noland 2012-11-17, 16:15
+
Dan Young 2012-11-17, 16:33
+
Dan Young 2012-11-19, 19:33
+
Patrick Wendell 2012-11-19, 23:03
+
Patrick Wendell 2012-11-19, 23:04
+
Brock Noland 2012-11-19, 23:29
+
Brock Noland 2012-11-20, 12:25
+
Dan Young 2012-11-20, 15:02
+
Brock Noland 2012-11-20, 16:21
+
Dan Young 2012-11-20, 16:59
+
Brock Noland 2012-11-20, 17:01
+
Dan Young 2012-11-20, 17:10
+
Brock Noland 2012-11-20, 17:14
+
Dan Young 2012-11-20, 17:17
+
Dan Young 2012-11-20, 20:03
+
Brock Noland 2012-11-20, 20:06
+
Patrick Wendell 2012-11-23, 12:46
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB