Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> [FLUME-1995] Remote Channel for Apache Flume


Copy link to this message
-
Re: [FLUME-1995] Remote Channel for Apache Flume
Hi Israel,
Some people deploy using RAID for that. We could also add software-level mirroring support to the FC. Why not do that?

Regards,
Mike

On May 22, 2013, at 2:08 PM, Israel Ekpo <[EMAIL PROTECTED]> wrote:

> I wanted to get some feedback from others before deciding whether or not to
> continue working on this.
>
> I initially filed this improvement/new feature because of use cases where
> there is a hardware failure on the machine where the agent is currently
> running.
>
> In terms of disaster recovery, having the events queue up on a remote
> machine (preferably in the same internal network) will allow another agent
> with the same configuration to pick it up from another machine and restart
> the process of data transport towards the sink.
>
> Sometimes, events may take a while to process and they may end up staying
> in the channels (FileChannel) for a long time, during which hardware
> failure could occur.
>
> If the data in the events is mission critical, this could cause a lot of
> headaches if there is no easy way to recover from the hardware failure
> after events have been queued up in the file channel.
>
> What are your thoughts towards the remote channel? I understand there is a
> JDBC Channel (http://flume.apache.org/FlumeUserGuide.html#jdbc-channel) but
> I have heard it has performance issues.
>
> This is why I am deciding to use a NoSQL store to solve this.
>
> I would like to get some feedback from others so that I can prioritize the
> tasks in my JIRA queue especially with the 1.4.0 release deadline drawing
> nearer.
>
> Thanks.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB