Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Spooling fileSuffix attribute ignored


+
ZORAIDA HIDALGO SANCHEZ 2013-05-21, 13:20
+
Phil Scala 2013-05-21, 15:17
+
Mike Percy 2013-05-22, 07:35
+
ZORAIDA HIDALGO SANCHEZ 2013-05-22, 07:43
Copy link to this message
-
Re: Spooling fileSuffix attribute ignored
You should check whether your version of Flume supports the ignorePattern
configuration param. The latest version on trunk does.

Mike

On Wed, May 22, 2013 at 12:43 AM, ZORAIDA HIDALGO SANCHEZ <[EMAIL PROTECTED]>wrote:

>  Oh! I see, then was a miss understanding. Ok, so we will need to find a
> workaround. Thanks a lot.
>
>   De: Mike Percy <[EMAIL PROTECTED]>
> Responder a: Flume User List <[EMAIL PROTECTED]>
> Fecha: miércoles, 22 de mayo de 2013 09:35
> Para: Flume User List <[EMAIL PROTECTED]>
> Asunto: Re: Spooling fileSuffix attribute ignored
>
>   Hi Phil,
> Nice approach. How is the spooling directory source working for you? Any
> thoughts on how it could be improved?
>
>  Mike
>
>
> On Tue, May 21, 2013 at 8:17 AM, Phil Scala <[EMAIL PROTECTED]>wrote:
>
>>  Hi,****
>>
>> ** **
>>
>> Based on my use and understanding that setting “fileSuffix” is simpy the
>> extension to the file to be added once the file was consumed and placed
>> onto the channel.  I don’t think it was intended to be used to indicate a
>> completely uploaded file.    In the dev newsgroup there was a discussion
>> about having the spooler ”wait” for a little while before ingesting the
>> file.  An ignore pattern may also be a good idea.****
>>
>> ** **
>>
>> Currently I am using an upload directory that I monitor and when lsof
>> reports a file is not in use/open I then move it to my spool directory.
>> ****
>>
>> ** **
>>
>> HTH****
>>
>> Phil****
>>
>> ** **
>>
>> ** **
>>
>> ** **
>>
>> Phil Scala****
>>
>> Software Developer / Architect
>> Global Relay
>>
>> [EMAIL PROTECTED]
>>
>> *866.484.6630*  |  [EMAIL PROTECTED]<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=mailto%3ainfo%40globalrelay.net>
>> |  globalrelay.com<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=http%3a%2f%2fwww.globalrelay.com%2f>
>>  ****
>>
>> ** **
>>
>> *From:* ZORAIDA HIDALGO SANCHEZ [mailto:[EMAIL PROTECTED]]
>> *Sent:* Tuesday, May 21, 2013 9:21 AM
>> *To:* Flume User List
>> *Subject:* Spooling fileSuffix attribute ignored****
>>
>> ** **
>>
>> Dear all, ****
>>
>> I am using the Spooling attribute "fileSuffix" to ensure that only when a
>> file has been completely upload into the Spooling dir,  is consumed by the
>> source. However, files with no suffix are also appended to the channel and
>> then processed by the sink. ****
>>
>> My configuration:****
>>
>> ** **
>>
>> tier1.sources  = s1****
>>
>> tier1.channels = c1****
>>
>> tier1.sinks    = s1****
>>
>> ** **
>>
>> # For each source, channel, and sink, set****
>>
>> # standard properties.****
>>
>> tier1.sources.s1.type     = spooldir****
>>
>> tier1.sources.s1.spoolDir = /home/user/flume/data****
>>
>> tier1.sources.s1.deletePolicy = immediate****
>>
>> tier1.sources.s1.batchSize = 1000****
>>
>> tier1.sources.s1.bufferMaxLines = 3000****
>>
>> tier1.sources.s1.fileHeader = true****
>>
>> tier1.sources.s1.fileSuffix=.COMPLETED****
>>
>> ** **
>>
>> Is that ok? ****
>>
>> ** **
>>
>> Thanks.****
>>
>> ** **
>>  ------------------------------
>>
>>
>> Este mensaje se dirige exclusivamente a su destinatario. Puede consultar
>> nuestra política de envío y recepción de correo electrónico en el enlace
>> situado más abajo.
>> This message is intended exclusively for its addressee. We only send and
>> receive email on the basis of the terms set out at:
>> http://www.tid.es/ES/PAGINAS/disclaimer.aspx****
>>
>
>
> ------------------------------
>
> Este mensaje se dirige exclusivamente a su destinatario. Puede consultar
> nuestra política de envío y recepción de correo electrónico en el enlace
> situado más abajo.
> This message is intended exclusively for its addressee. We only send and
> receive email on the basis of the terms set out at:
> http://www.tid.es/ES/PAGINAS/disclaimer.aspx
>
+
Phil Scala 2013-05-22, 22:55
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB