Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Spooling fileSuffix attribute ignored


+
ZORAIDA HIDALGO SANCHEZ 2013-05-21, 13:20
+
Phil Scala 2013-05-21, 15:17
+
Mike Percy 2013-05-22, 07:35
+
ZORAIDA HIDALGO SANCHEZ 2013-05-22, 07:43
Copy link to this message
-
Re: Spooling fileSuffix attribute ignored
Mike Percy 2013-05-22, 07:59
You should check whether your version of Flume supports the ignorePattern
configuration param. The latest version on trunk does.

Mike

On Wed, May 22, 2013 at 12:43 AM, ZORAIDA HIDALGO SANCHEZ <[EMAIL PROTECTED]>wrote:

>  Oh! I see, then was a miss understanding. Ok, so we will need to find a
> workaround. Thanks a lot.
>
>   De: Mike Percy <[EMAIL PROTECTED]>
> Responder a: Flume User List <[EMAIL PROTECTED]>
> Fecha: miércoles, 22 de mayo de 2013 09:35
> Para: Flume User List <[EMAIL PROTECTED]>
> Asunto: Re: Spooling fileSuffix attribute ignored
>
>   Hi Phil,
> Nice approach. How is the spooling directory source working for you? Any
> thoughts on how it could be improved?
>
>  Mike
>
>
> On Tue, May 21, 2013 at 8:17 AM, Phil Scala <[EMAIL PROTECTED]>wrote:
>
>>  Hi,****
>>
>> ** **
>>
>> Based on my use and understanding that setting “fileSuffix” is simpy the
>> extension to the file to be added once the file was consumed and placed
>> onto the channel.  I don’t think it was intended to be used to indicate a
>> completely uploaded file.    In the dev newsgroup there was a discussion
>> about having the spooler ”wait” for a little while before ingesting the
>> file.  An ignore pattern may also be a good idea.****
>>
>> ** **
>>
>> Currently I am using an upload directory that I monitor and when lsof
>> reports a file is not in use/open I then move it to my spool directory.
>> ****
>>
>> ** **
>>
>> HTH****
>>
>> Phil****
>>
>> ** **
>>
>> ** **
>>
>> ** **
>>
>> Phil Scala****
>>
>> Software Developer / Architect
>> Global Relay
>>
>> [EMAIL PROTECTED]
>>
>> *866.484.6630*  |  [EMAIL PROTECTED]<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=mailto%3ainfo%40globalrelay.net>
>> |  globalrelay.com<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=http%3a%2f%2fwww.globalrelay.com%2f>
>>  ****
>>
>> ** **
>>
>> *From:* ZORAIDA HIDALGO SANCHEZ [mailto:[EMAIL PROTECTED]]
>> *Sent:* Tuesday, May 21, 2013 9:21 AM
>> *To:* Flume User List
>> *Subject:* Spooling fileSuffix attribute ignored****
>>
>> ** **
>>
>> Dear all, ****
>>
>> I am using the Spooling attribute "fileSuffix" to ensure that only when a
>> file has been completely upload into the Spooling dir,  is consumed by the
>> source. However, files with no suffix are also appended to the channel and
>> then processed by the sink. ****
>>
>> My configuration:****
>>
>> ** **
>>
>> tier1.sources  = s1****
>>
>> tier1.channels = c1****
>>
>> tier1.sinks    = s1****
>>
>> ** **
>>
>> # For each source, channel, and sink, set****
>>
>> # standard properties.****
>>
>> tier1.sources.s1.type     = spooldir****
>>
>> tier1.sources.s1.spoolDir = /home/user/flume/data****
>>
>> tier1.sources.s1.deletePolicy = immediate****
>>
>> tier1.sources.s1.batchSize = 1000****
>>
>> tier1.sources.s1.bufferMaxLines = 3000****
>>
>> tier1.sources.s1.fileHeader = true****
>>
>> tier1.sources.s1.fileSuffix=.COMPLETED****
>>
>> ** **
>>
>> Is that ok? ****
>>
>> ** **
>>
>> Thanks.****
>>
>> ** **
>>  ------------------------------
>>
>>
>> Este mensaje se dirige exclusivamente a su destinatario. Puede consultar
>> nuestra política de envío y recepción de correo electrónico en el enlace
>> situado más abajo.
>> This message is intended exclusively for its addressee. We only send and
>> receive email on the basis of the terms set out at:
>> http://www.tid.es/ES/PAGINAS/disclaimer.aspx****
>>
>
>
> ------------------------------
>
> Este mensaje se dirige exclusivamente a su destinatario. Puede consultar
> nuestra política de envío y recepción de correo electrónico en el enlace
> situado más abajo.
> This message is intended exclusively for its addressee. We only send and
> receive email on the basis of the terms set out at:
> http://www.tid.es/ES/PAGINAS/disclaimer.aspx
>
+
Phil Scala 2013-05-22, 22:55