Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume is not adding .tmp suffix for in-progress files


Copy link to this message
-
Re: Flume is not adding .tmp suffix for in-progress files
“.tmp” is added only for the HDFS sink, not the rolling file sink.  
Thanks,
Hari
On Friday, October 25, 2013 at 1:43 PM, Lenin Raj wrote:

> Hi All,
>  
> I am having a simple agent config like below:
>  
> agent.sources = Twitter
> agent.channels = MemChannel
> agent.sinks = FileSink
>  
> agent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
> agent.sources.Twitter.channels = MemChannel
>  
> agent.sinks.FileSink.channel = MemChannel
> agent.sinks.FileSink.type = file_roll
> agent.sinks.FileSink.sink.directory = /var/data/
> agent.sinks.FileSink.sink.rollInterval = 60
>  
> agent.channels.MemChannel.type = memory
> agent.channels.MemChannel.capacity = 50000
> agent.channels.MemChannel.transactionCapacity = 1000
>  
>  
> root@sys:/var/data# ls -ltrh
> total 8.0K
> -rw-r--r-- 1 root root 4.9K Oct 26 02:00 1382733013173-1
> -rw-r--r-- 1 root root    0 Oct 26 02:01 1382733013173-2
> root@ sys:/var/data# ls -ltrh
> total 12K
> -rw-r--r-- 1 root root 4.9K Oct 26 02:00 1382733013173-1
> -rw-r--r-- 1 root root 2.1K Oct 26 02:01 1382733013173-2
>  
> As you can see, the file "1382733013173-2" is being modified without the ".tmp" prefix.
>  
> My Flume version:
> Flume 1.4.0
> Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
>  
>  
> Does the config file need any modifications to get .tmp suffix? I need this suffix to exclude this file from my file watcher script.
>  
> Thanks,
> Lenin