Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Multiple Mirror Architecture Example


Copy link to this message
-
Re: Multiple Mirror Architecture Example
Unfortunately, no - the mirroring mechanism needs ZK to function. I think
some people have used plain rsync's to maintain mirrors, but if there is a
strong case for this then we can think about implementing a less
full-fledged mirroring component that uses SimpleConsumer and talks to the
brokers directly without using ZK.

Joel
On Thu, Jul 26, 2012 at 11:49 AM, Christian Carollo <[EMAIL PROTECTED]>wrote:

> This configuration can be done with out ZK, correct?
>
> On Jul 26, 2012, at 11:43 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:
>
> > I see - yes that should be fine. i.e., as long as the topics are distinct
> > then you can mirror both ways and the topic "owned" by each machine
> should
> > be read-only on the other machine.
> >
> > Thanks,
> >
> > Joel
> >
> > On Thu, Jul 26, 2012 at 11:38 AM, Christian Carollo <[EMAIL PROTECTED]
> >wrote:
> >
> >> Hi Joel,
> >>
> >> Thanks for getting back to me.  I actually missed your response.
> >>
> >> What I really am trying to is…
> >>
> >> Have two machines each running a single broker (not clustered)?  One
> topic
> >> is homed on machine A
> >> an mirrored to machine B and another topic is homed on machine B and
> >> mirrored to machine A.
> >>
> >> When machine A goes down machine B can take over as owner of the topic
> >> that machine A was previously owner of.
> >>
> >> Is that clearer?
> >>
> >> Thanks
> >> Christian
> >>
> >> On Jul 18, 2012, at 10:21 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:
> >>
> >>> Missed this thread - can you elaborate a bit more on your use case? I'm
> >>> assuming you are referring to the inter-cluster (mirroring) feature
> (and
> >>> not replication). It isn't really possible for two clusters to mirror
> >> each
> >>> other - as the mirroring would never end, at least the way it is
> >>> implemented right now. If you want the aggregate data from all your
> >>> data-centers to be available in each data-center there are some
> >> topologies
> >>> that may work for you.
> >>>
> >>> Joel
> >>>
> >>> On Mon, Jul 16, 2012 at 2:32 PM, Christian Carollo <[EMAIL PROTECTED]
> >>> wrote:
> >>>
> >>>> We are looking to implement a multi-mirror queue architecture and
> >>>> wondering if there are any known documents or examples.  In general,
> we
> >> are
> >>>> looking to setup two Kafka clusters that replicate or mirror one
> >> another.
> >>>>
> >>>> Christian
> >>
> >>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB