Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Expected behavior for clustered kafka setup


Copy link to this message
-
Re: Expected behavior for clustered kafka setup
Jun Rao 2013-04-24, 01:06
It doesn't really matter, but bringing ZK up first is probably better.

Thanks,

Jun
On Tue, Apr 23, 2013 at 12:14 PM, Karl Kirch <[EMAIL PROTECTED]> wrote:

> Thanks,
>
> I'll try to keep an eye on that and see if that fixes the problem.
> Also, how will that affect things if I have a node completely go away
> (i.e. zookeeper and kafka both go offline for a single node).
> Should I still bring up zookeeper on that node first or does it matter at
> that point?
>
>
> Karl
>
>
> On Apr 23, 2013, at 2:11 PM, Xavier Stevens <[EMAIL PROTECTED]>
>  wrote:
>
> > You should bring up your Zookeeper instances first and then the Kafka
> > brokers.
> >
> >
> > On Tue, Apr 23, 2013 at 11:56 AM, Karl Kirch <[EMAIL PROTECTED]> wrote:
> >
> >> Now to make things even more interesting. I restarted 2 and now it sees
> >> all 3 nodes.
> >> I think I've got some sort of weirdness happening with how I'm bringing
> >> everything online.
> >>
> >> I've tried starting up all the zookeeper instances and then each of the
> >> brokers. I've tried starting up all of the kafka brokers and then the
> >> zookeeper instances. I've also tried bringing up each machine (zookeeper
> >> and kafka) up individually. All seem to give inconsistent results.
> >>
> >> Is there a recommended way to bring nodes online?
> >>
> >> Karl
> >>
> >> On Apr 23, 2013, at 1:36 PM, Karl Kirch <[EMAIL PROTECTED]> wrote:
> >>
> >>> I've tried doing what was suggested (creating the topic folders and
> >> restarting) and I'm getting slightly different but still bad results.
> >>> Now it'll write to node 1 and 3 but not to 2.
> >>>
> >>> Karl
> >>>
> >>> On Apr 23, 2013, at 1:27 PM, Neha Narkhede <[EMAIL PROTECTED]>
> >>> wrote:
> >>>
> >>>> Can you see if you are running to
> >>>> https://issues.apache.org/jira/browse/KAFKA-278 ?
> >>>>
> >>>> Thanks,
> >>>> Neha
> >>>>
> >>>> On Tue, Apr 23, 2013 at 11:24 AM, Karl Kirch <[EMAIL PROTECTED]>
> wrote:
> >>>>> Zookeeper based.
> >>>>>
> >>>>> Karl
> >>>>>
> >>>>> On Apr 23, 2013, at 1:15 PM, Neha Narkhede <[EMAIL PROTECTED]>
> >>>>> wrote:
> >>>>>
> >>>>>> What kind of producer do you use ? Zookeeper based or broker list ?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Neha
> >>>>>>
> >>>>>> On Tue, Apr 23, 2013 at 9:02 AM, Karl Kirch <[EMAIL PROTECTED]>
> >> wrote:
> >>>>>>> For some extra info, this was a restart of the just zookeeper and
> >> kafka, our clients were all still online.
> >>>>>>>
> >>>>>>> Karl
> >>>>>>>
> >>>>>>> On Apr 23, 2013, at 10:50 AM, Karl Kirch <[EMAIL PROTECTED]>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10
> >> partitions per node) and Zookeeper 3.3.4 on each node. I'm seeing that
> only
> >> one broker is being used to actually send messages through. We're seeing
> >> this behavior after a restart and deletion of the kafka data
> >> (/tmp/kafka-logs). In the past its used all three and worked fine.
> >>>>>>>>
> >>>>>>>> Is this normal? (i.e. it takes some time to spread across the
> >> cluster) Or should it distribute stuff immediately?
> >>>>>>>>
> >>>>>>>> Karl
> >>>>>>>
> >>>>>
> >>>
> >>
> >>
>
>