Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - File Sink/Source

Copy link to this message
File Sink/Source
David Sinclair 2013-10-04, 18:59

I have a question regarding the RollingFileSink and
SpoolingDirectorySource. I was trying to write everything from an AMQP
source to a file sink, then have the spooling directory source pick up
these files. This won't work as the files aren't immutable.

If I use a File Channel to store the events between my source and sink, is
there a concern about the number of events in the channel if the sink is
unable to deliver said events? For example, I will be getting around 5K
messages/sec and the size is about 2K. So roughly 10MB a second. If the
sink is unable to deliver the messages for 2 hours, that would be 36
million events in the channel.

Is the file channel designed to handle this? Or should I have a file sink
in between.