Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Remote consumer

Copy link to this message
Re: Remote consumer
One option would be to use Cassandra for the multi data center replication
and have Kafka Consumes update the Cassandra ring in each data center.

This allows you to have active / active / active applications.  It also
allows you to choose a ring and map/reduce the data out of it (like with
Pig) which will saturate that ring so you want to-do it in an availability
zone where your application is not serving out of.

Every data center has its own "commit log" thanks to the Kafka broker
cluster and data center replication from a database with a Cassandra ring.

This is the setup I have used before and continue to work on now.

I have been meaning to start open sourcing some of this (it uses Thrift
On Thu, Oct 31, 2013 at 8:37 AM, Muhzin <[EMAIL PROTECTED]> wrote:

> Hi,
> We are architecting an  application in AWS with  multi region deployment
>  and were  evaluating kafka as the messaging system for  data replication
> and activity tracking.
> The initial idea is to have the producer and broker cluster to reside in
> Ireland. We wanted to know the feasibility of having a remote consumer in
> Singapore.
> Found a link to improve throughput for a remote consumer -
> http://bit.ly/1cp7T6Z
> #) Has anybody done something similar - if so, could you please shed some
> light on how latency will play and some of the best practices that could be
> done.
> #) What would be a good way to secure the communication if it is happening
> over the internet. We were thinking about SSL tunnel (using OpenVPN) as
> described in  http://aws.amazon.com/articles/0639686206802544. Thoughts?
> --
> BR
> Muhsin