Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> File Sink/Source


Copy link to this message
-
Re: File Sink/Source
Have you tried the JMS Source? It can pick up data directly into Flume.
Thanks,
Hari
On Friday, October 4, 2013 at 11:59 AM, David Sinclair wrote:

> Hi,
>
> I have a question regarding the RollingFileSink and SpoolingDirectorySource. I was trying to write everything from an AMQP source to a file sink, then have the spooling directory source pick up these files. This won't work as the files aren't immutable.  
>
> If I use a File Channel to store the events between my source and sink, is there a concern about the number of events in the channel if the sink is unable to deliver said events? For example, I will be getting around 5K messages/sec and the size is about 2K. So roughly 10MB a second. If the sink is unable to deliver the messages for 2 hours, that would be 36 million events in the channel.
>
> Is the file channel designed to handle this? Or should I have a file sink in between.
>
> thanks
>
> dave