Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Kafka 0.8 cluster setup?


+
Jason Huang 2013-04-22, 21:13
+
Neha Narkhede 2013-04-22, 21:32
+
Chris Curtin 2013-04-23, 12:29
+
Jason Huang 2013-04-23, 13:51
+
Eric Sites 2013-04-23, 14:22
+
Jason Huang 2013-04-23, 14:31
Copy link to this message
-
Re: Kafka 0.8 cluster setup?
Chris Curtin 2013-04-23, 14:42
Beat me to it ;)

Only caveat is I wouldn't use /tmp for this, since if you're running
tmpwatch the 'myid' file will get removed unexpectedly since it doesn't
seem to be changed at it. That was fun to find :)

We use /var/zookeeper for our storage.

Thanks,

Chris
On Tue, Apr 23, 2013 at 10:30 AM, Jason Huang <[EMAIL PROTECTED]> wrote:

> Thanks Eric - this helps quite a bit.
>
> I will play around with it.
>
> Jason
>
> On Tue, Apr 23, 2013 at 10:21 AM, Eric Sites <[EMAIL PROTECTED]
> >wrote:
>
> > Jason,
> >
> > You need to modify the ZooKeeper config and add the following:
> >
> > dataDir=/tmp/zookeeper
> >
> > initLimit=50
> > syncLimit=2
> >
> > server.1=kafka001.domain.com:2888:3888
> > server.2=kafka002.domain.com:2888:3888
> >
> > server.3=kafka003.domain.com:2888:3888
> >
> >
> >
> > # Make sure you open those 2 points on each of the servers for your
> > firewall
> >
> > The .1 after the server is the id of the zookeeper which should be stored
> > in a file
> > Named /tmp/zookeeper/myid
> >
> > Echo "1" > /tmp/zookeeper/myid`
> >
> >
> > Cheers,
> > Eric Sites
> >
> >
> > On 4/23/13 9:50 AM, "Jason Huang" <[EMAIL PROTECTED]> wrote:
> >
> > >Thanks Chris and Neha.
> > >
> > >Chris - I've been through the link you mentioned before. However, that
> > >appears to be using one instance of zookeeper, which makes whichever
> > >server
> > >that runs zookeeper as the single point of failure?
> > >
> > >Jason
> > >
> > >On Tue, Apr 23, 2013 at 8:28 AM, Chris Curtin
> > ><[EMAIL PROTECTED]>wrote:
> > >
> > >> I following these instructions to get the first 'play' cluster going:
> > >>
> > >> https://cwiki.apache.org/KAFKA/kafka-08-quick-start.html
> > >>
> > >> Instead of running the 3 brokers on the same machine, I ran on on each
> > >> machine.
> > >>
> > >> Note that you will need to do a little bit of zookeeper setup to get a
> > >> cluster running, I think the main one was setting the 'myid' in the
> > >> zookeeper install directory properly on each machine (namely give each
> > >> machine a unique # in that file).
> > >>
> > >> Thanks,
> > >>
> > >> Chris
> > >>
> > >>
> > >> On Mon, Apr 22, 2013 at 5:32 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > >> >wrote:
> > >>
> > >> > Hi Jason,
> > >> >
> > >> > We are in the process of updating the documentation. Hoping to
> finish
> > >> > it by this week. Stay tuned.
> > >> >
> > >> > Thanks,
> > >> > Neha
> > >> >
> > >> > On Mon, Apr 22, 2013 at 2:12 PM, Jason Huang <[EMAIL PROTECTED]
> >
> > >> > wrote:
> > >> > > Hello,
> > >> > >
> > >> > > We've been playing around with kafka 0.8 for a few months now and
> > >> decided
> > >> > > to install kafka on a small cluster for further testing. I tried
> to
> > >> > search
> > >> > > online but couldn't find any setup documentation for a kafka 0.8
> > >> cluster.
> > >> > >
> > >> > > Does anyone know if such documents exist? If they don't exist,
> what
> > >> could
> > >> > > be our best guide to start with?
> > >> > >
> > >> > > thanks!
> > >> > >
> > >> > > Jason
> > >> >
> > >>
> >
> >
>

 
+
Jason Huang 2013-04-23, 14:49
+
Jun Rao 2013-04-23, 15:43