Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - .SpoolingFileLineReader warning....


Copy link to this message
-
Re: .SpoolingFileLineReader warning....
Dan Young 2012-11-17, 16:33
logrotate config, in /etc/logrotate.d, ran from cron.hourly.

/var/log/clickstream/clickstream.log
{
  missingok
  rotate 3
  compress
  delaycompress
  copytruncate
  notifempty
  size 50M
  dateext
  dateformat -%Y-%m-%d-%s
  create 666 ubuntu ubuntu
  postrotate
  cp -p $1 /mnt/flume/clickstream/ 2>&1
  endscript
}

flume config:

# Name the components on this agent
agent1.sources = c1
agent1.sinks = c1s3
agent1.channels = ch1

# Describe/configure
agent1.sources.c1.type = org.apache.flume.source.SpoolDirectorySource
agent1.sources.c1.spoolDir = /mnt/flume/clickstream
agent1.sources.c1.fileHeader = false
agent1.sources.c1.interceptors = a b
agent1.sources.c1.interceptors.a.type org.apache.flume.interceptor.TimestampInterceptor$Builder
agent1.sources.c1.interceptors.b.type org.apache.flume.interceptor.HostInterceptor$Builder
agent1.sources.c1.interceptors.b.preserveExisting = false
agent1.sources.c1.interceptors.b.hostHeader = host
# Describe s3
agent1.sinks.c1s3.type = hdfs
agent1.sinks.c1s3.hdfs.path s3n://<super_secret_stuff_here>@<my_bucket>/clicks/%Y/%m
agent1.sinks.c1s3.hdfs.rollInterval = 300
agent1.sinks.c1s3.hdfs.rollSize = 0
agent1.sinks.c1s3.hdfs.rollCount = 0
agent1.sinks.c1s3.hdfs.batchSize = 400000
agent1.sinks.c1s3.hdfs.codeC = gzip
agent1.sinks.c1s3.hdfs.fileType = CompressedStream
agent1.sinks.c1s3.hdfs.writeFormat = Text
agent1.sinks.c1s3.hdfs.filePrefix = clicks-%Y-%m-%d-%H-%M-%{host}-
agent1.sinks.c1s3.hdfs.round = true
agent1.sinks.c1s3.hdfs.roundValue = 10
agent1.sinks.c1s3.hdfs.roundUnit = minute

# Use a channel which buffers events in memory
agent1.channels.ch1.type = file
agent1.channels.ch1.transactionCapacity = 400000
agent1.channels.ch1.capacity = 2000000
agent1.channels.ch1.checkpointDir = /mnt/flume/.flume/file-ch1/checkpoint
agent1.channels.ch1.dataDirs = /mnt/flume/.flume/file-ch1/data
agent1.channels.ch1.checkpointInterval = 30000
# Bind the source and sink to the channel
agent1.sources.c1.channels = ch1
agent1.sinks.c1s3.channel = ch1

On Sat, Nov 17, 2012 at 9:15 AM, Brock Noland <[EMAIL PROTECTED]> wrote:

>  Ok, do you mind sharing your log rotate config to see if we can
> reproduce?
>
> --
> Brock Noland
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> On Saturday, November 17, 2012 at 10:01 AM, Dan Young wrote:
>
> Hey Brock,
>
> No I have not modified the conf while the agent was running.
>
> /mnt/flume is local. Note that this is running on an ec2 instance and the
> disk is the ephemeral drive, not EBS.
>
> Regards ,
>
> Dano
> On Nov 17, 2012 8:58 AM, "Brock Noland" <[EMAIL PROTECTED]> wrote:
>
> Hi,
>
> I highly doubt it's related to
> (https://issues.apache.org/jira/browse/FLUME-1721) but have you
> modified the configuration file since starting the agent?  If so, can
> you restart the agent and see if the error continues?
>
> Also, is /mnt/flume local disk or NAS?
>
> Brock
>
> On Sat, Nov 17, 2012 at 9:02 AM, Dan Young <[EMAIL PROTECTED]> wrote:
> > First a bit of context, I'm using logrotate to monitor and copy (cp -p)
> log
> > files to a flume spooling directory source.  So every hour, logrotate
> checks
> > for and copies a file from the source to the flume destination. I see the
> > following warning message in the flume logs.
> >
> >
> > 17 Nov 2012 14:47:07,682 WARN  [pool-10-thread-1]
> > (org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile:328)  -
> > Could not find file:
> > /mnt/flume/clickstream/clickstream.log-2012-11-17-1353163623
> > java.io.FileNotFoundException:
> > /mnt/flume/clickstream/clickstream.log-2012-11-17-1353163623 (Permission
> > denied)
> > at java.io.FileInputStream.open(Native Method)
> > at java.io.FileInputStream.<init>(FileInputStream.java:138)
> > at java.io.FileReader.<init>(FileReader.java:72)
> > at
> >
> org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFileLineReader.java:322)
> > at
> >
> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:172)