At what point would your business logic decide that the file is complete (by time or other decision to cut a file as completed)? And then when do you batch process from what the stream has pilled up for you ?
Whether you need more layers and overhead all gets back to what you are trying to accomplish and such :) You might need to use Zookeeper or something to coordinate what to run the batch process (depending on how you kick this off) so you know what is going on in the Consumers is completed in the other system.
On Fri, Aug 30, 2013 at 2:18 PM, Mark <[EMAIL PROTECTED]> wrote:
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext