Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume is not adding .tmp suffix for in-progress files


Copy link to this message
-
Re: Flume is not adding .tmp suffix for in-progress files
“.tmp” is added only for the HDFS sink, not the rolling file sink.  
Thanks,
Hari
On Friday, October 25, 2013 at 1:43 PM, Lenin Raj wrote:

> Hi All,
>  
> I am having a simple agent config like below:
>  
> agent.sources = Twitter
> agent.channels = MemChannel
> agent.sinks = FileSink
>  
> agent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
> agent.sources.Twitter.channels = MemChannel
>  
> agent.sinks.FileSink.channel = MemChannel
> agent.sinks.FileSink.type = file_roll
> agent.sinks.FileSink.sink.directory = /var/data/
> agent.sinks.FileSink.sink.rollInterval = 60
>  
> agent.channels.MemChannel.type = memory
> agent.channels.MemChannel.capacity = 50000
> agent.channels.MemChannel.transactionCapacity = 1000
>  
>  
> root@sys:/var/data# ls -ltrh
> total 8.0K
> -rw-r--r-- 1 root root 4.9K Oct 26 02:00 1382733013173-1
> -rw-r--r-- 1 root root    0 Oct 26 02:01 1382733013173-2
> root@ sys:/var/data# ls -ltrh
> total 12K
> -rw-r--r-- 1 root root 4.9K Oct 26 02:00 1382733013173-1
> -rw-r--r-- 1 root root 2.1K Oct 26 02:01 1382733013173-2
>  
> As you can see, the file "1382733013173-2" is being modified without the ".tmp" prefix.
>  
> My Flume version:
> Flume 1.4.0
> Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
>  
>  
> Does the config file need any modifications to get .tmp suffix? I need this suffix to exclude this file from my file watcher script.
>  
> Thanks,
> Lenin

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB