Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Multiple Mirror Architecture Example


Copy link to this message
-
Re: Multiple Mirror Architecture Example
Joel Koshy 2012-07-26, 18:52
Unfortunately, no - the mirroring mechanism needs ZK to function. I think
some people have used plain rsync's to maintain mirrors, but if there is a
strong case for this then we can think about implementing a less
full-fledged mirroring component that uses SimpleConsumer and talks to the
brokers directly without using ZK.

Joel
On Thu, Jul 26, 2012 at 11:49 AM, Christian Carollo <[EMAIL PROTECTED]>wrote:

> This configuration can be done with out ZK, correct?
>
> On Jul 26, 2012, at 11:43 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:
>
> > I see - yes that should be fine. i.e., as long as the topics are distinct
> > then you can mirror both ways and the topic "owned" by each machine
> should
> > be read-only on the other machine.
> >
> > Thanks,
> >
> > Joel
> >
> > On Thu, Jul 26, 2012 at 11:38 AM, Christian Carollo <[EMAIL PROTECTED]
> >wrote:
> >
> >> Hi Joel,
> >>
> >> Thanks for getting back to me.  I actually missed your response.
> >>
> >> What I really am trying to is…
> >>
> >> Have two machines each running a single broker (not clustered)?  One
> topic
> >> is homed on machine A
> >> an mirrored to machine B and another topic is homed on machine B and
> >> mirrored to machine A.
> >>
> >> When machine A goes down machine B can take over as owner of the topic
> >> that machine A was previously owner of.
> >>
> >> Is that clearer?
> >>
> >> Thanks
> >> Christian
> >>
> >> On Jul 18, 2012, at 10:21 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:
> >>
> >>> Missed this thread - can you elaborate a bit more on your use case? I'm
> >>> assuming you are referring to the inter-cluster (mirroring) feature
> (and
> >>> not replication). It isn't really possible for two clusters to mirror
> >> each
> >>> other - as the mirroring would never end, at least the way it is
> >>> implemented right now. If you want the aggregate data from all your
> >>> data-centers to be available in each data-center there are some
> >> topologies
> >>> that may work for you.
> >>>
> >>> Joel
> >>>
> >>> On Mon, Jul 16, 2012 at 2:32 PM, Christian Carollo <[EMAIL PROTECTED]
> >>> wrote:
> >>>
> >>>> We are looking to implement a multi-mirror queue architecture and
> >>>> wondering if there are any known documents or examples.  In general,
> we
> >> are
> >>>> looking to setup two Kafka clusters that replicate or mirror one
> >> another.
> >>>>
> >>>> Christian
> >>
> >>
>
>