Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> File Sink/Source


Copy link to this message
-
File Sink/Source
Hi,

I have a question regarding the RollingFileSink and
SpoolingDirectorySource. I was trying to write everything from an AMQP
source to a file sink, then have the spooling directory source pick up
these files. This won't work as the files aren't immutable.

If I use a File Channel to store the events between my source and sink, is
there a concern about the number of events in the channel if the sink is
unable to deliver said events? For example, I will be getting around 5K
messages/sec and the size is about 2K. So roughly 10MB a second. If the
sink is unable to deliver the messages for 2 hours, that would be 36
million events in the channel.

Is the file channel designed to handle this? Or should I have a file sink
in between.

thanks

dave
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB