Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Spooling fileSuffix attribute ignored


+
ZORAIDA HIDALGO SANCHEZ 2013-05-21, 13:20
+
Phil Scala 2013-05-21, 15:17
Copy link to this message
-
Re: Spooling fileSuffix attribute ignored
Mike Percy 2013-05-22, 07:35
Hi Phil,
Nice approach. How is the spooling directory source working for you? Any
thoughts on how it could be improved?

Mike
On Tue, May 21, 2013 at 8:17 AM, Phil Scala <[EMAIL PROTECTED]>wrote:

> Hi,****
>
> ** **
>
> Based on my use and understanding that setting “fileSuffix” is simpy the
> extension to the file to be added once the file was consumed and placed
> onto the channel.  I don’t think it was intended to be used to indicate a
> completely uploaded file.    In the dev newsgroup there was a discussion
> about having the spooler ”wait” for a little while before ingesting the
> file.  An ignore pattern may also be a good idea.****
>
> ** **
>
> Currently I am using an upload directory that I monitor and when lsof
> reports a file is not in use/open I then move it to my spool directory.  *
> ***
>
> ** **
>
> HTH****
>
> Phil****
>
> ** **
>
> ** **
>
> ** **
>
> Phil Scala****
>
> Software Developer / Architect
> Global Relay
>
> [EMAIL PROTECTED]
>
> *866.484.6630*  |  [EMAIL PROTECTED]<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=mailto%3ainfo%40globalrelay.net>
> |  globalrelay.com<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=http%3a%2f%2fwww.globalrelay.com%2f>
>  ****
>
> ** **
>
> *From:* ZORAIDA HIDALGO SANCHEZ [mailto:[EMAIL PROTECTED]]
> *Sent:* Tuesday, May 21, 2013 9:21 AM
> *To:* Flume User List
> *Subject:* Spooling fileSuffix attribute ignored****
>
> ** **
>
> Dear all, ****
>
> I am using the Spooling attribute "fileSuffix" to ensure that only when a
> file has been completely upload into the Spooling dir,  is consumed by the
> source. However, files with no suffix are also appended to the channel and
> then processed by the sink. ****
>
> My configuration:****
>
> ** **
>
> tier1.sources  = s1****
>
> tier1.channels = c1****
>
> tier1.sinks    = s1****
>
> ** **
>
> # For each source, channel, and sink, set****
>
> # standard properties.****
>
> tier1.sources.s1.type     = spooldir****
>
> tier1.sources.s1.spoolDir = /home/user/flume/data****
>
> tier1.sources.s1.deletePolicy = immediate****
>
> tier1.sources.s1.batchSize = 1000****
>
> tier1.sources.s1.bufferMaxLines = 3000****
>
> tier1.sources.s1.fileHeader = true****
>
> tier1.sources.s1.fileSuffix=.COMPLETED****
>
> ** **
>
> Is that ok? ****
>
> ** **
>
> Thanks.****
>
> ** **
> ------------------------------
>
>
> Este mensaje se dirige exclusivamente a su destinatario. Puede consultar
> nuestra política de envío y recepción de correo electrónico en el enlace
> situado más abajo.
> This message is intended exclusively for its addressee. We only send and
> receive email on the basis of the terms set out at:
> http://www.tid.es/ES/PAGINAS/disclaimer.aspx****
>
+
ZORAIDA HIDALGO SANCHEZ 2013-05-22, 07:43
+
Mike Percy 2013-05-22, 07:59
+
Phil Scala 2013-05-22, 22:55