Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> File Sink/Source


Copy link to this message
-
Re: File Sink/Source
Have you tried the JMS Source? It can pick up data directly into Flume.
Thanks,
Hari
On Friday, October 4, 2013 at 11:59 AM, David Sinclair wrote:

> Hi,
>
> I have a question regarding the RollingFileSink and SpoolingDirectorySource. I was trying to write everything from an AMQP source to a file sink, then have the spooling directory source pick up these files. This won't work as the files aren't immutable.  
>
> If I use a File Channel to store the events between my source and sink, is there a concern about the number of events in the channel if the sink is unable to deliver said events? For example, I will be getting around 5K messages/sec and the size is about 2K. So roughly 10MB a second. If the sink is unable to deliver the messages for 2 hours, that would be 36 million events in the channel.
>
> Is the file channel designed to handle this? Or should I have a file sink in between.
>
> thanks
>
> dave

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB