Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Spooling fileSuffix attribute ignored

Copy link to this message
RE: Spooling fileSuffix attribute ignored

Based on my use and understanding that setting "fileSuffix" is simpy the extension to the file to be added once the file was consumed and placed onto the channel.  I don't think it was intended to be used to indicate a completely uploaded file.    In the dev newsgroup there was a discussion about having the spooler "wait" for a little while before ingesting the file.  An ignore pattern may also be a good idea.

Currently I am using an upload directory that I monitor and when lsof reports a file is not in use/open I then move it to my spool directory.


Phil Scala
Software Developer / Architect
Global Relay


866.484.6630  |  [EMAIL PROTECTED]<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=mailto%3ainfo%40globalrelay.net>  |  globalrelay.com<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=http%3a%2f%2fwww.globalrelay.com%2f>

Sent: Tuesday, May 21, 2013 9:21 AM
To: Flume User List
Subject: Spooling fileSuffix attribute ignored

Dear all,
I am using the Spooling attribute "fileSuffix" to ensure that only when a file has been completely upload into the Spooling dir,  is consumed by the source. However, files with no suffix are also appended to the channel and then processed by the sink.
My configuration:

tier1.sources  = s1
tier1.channels = c1
tier1.sinks    = s1

# For each source, channel, and sink, set
# standard properties.
tier1.sources.s1.type     = spooldir
tier1.sources.s1.spoolDir = /home/user/flume/data
tier1.sources.s1.deletePolicy = immediate
tier1.sources.s1.batchSize = 1000
tier1.sources.s1.bufferMaxLines = 3000
tier1.sources.s1.fileHeader = true

Is that ok?



Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at: