Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Kafka 0.8 cluster setup?


Copy link to this message
-
Re: Kafka 0.8 cluster setup?
Jason Huang 2013-04-23, 14:49
Good point.

thanks!

Jason

On Tue, Apr 23, 2013 at 10:42 AM, Chris Curtin <[EMAIL PROTECTED]>wrote:

> Beat me to it ;)
>
> Only caveat is I wouldn't use /tmp for this, since if you're running
> tmpwatch the 'myid' file will get removed unexpectedly since it doesn't
> seem to be changed at it. That was fun to find :)
>
> We use /var/zookeeper for our storage.
>
> Thanks,
>
> Chris
>
>
> On Tue, Apr 23, 2013 at 10:30 AM, Jason Huang <[EMAIL PROTECTED]>
> wrote:
>
> > Thanks Eric - this helps quite a bit.
> >
> > I will play around with it.
> >
> > Jason
> >
> > On Tue, Apr 23, 2013 at 10:21 AM, Eric Sites <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Jason,
> > >
> > > You need to modify the ZooKeeper config and add the following:
> > >
> > > dataDir=/tmp/zookeeper
> > >
> > > initLimit=50
> > > syncLimit=2
> > >
> > > server.1=kafka001.domain.com:2888:3888
> > > server.2=kafka002.domain.com:2888:3888
> > >
> > > server.3=kafka003.domain.com:2888:3888
> > >
> > >
> > >
> > > # Make sure you open those 2 points on each of the servers for your
> > > firewall
> > >
> > > The .1 after the server is the id of the zookeeper which should be
> stored
> > > in a file
> > > Named /tmp/zookeeper/myid
> > >
> > > Echo "1" > /tmp/zookeeper/myid`
> > >
> > >
> > > Cheers,
> > > Eric Sites
> > >
> > >
> > > On 4/23/13 9:50 AM, "Jason Huang" <[EMAIL PROTECTED]> wrote:
> > >
> > > >Thanks Chris and Neha.
> > > >
> > > >Chris - I've been through the link you mentioned before. However, that
> > > >appears to be using one instance of zookeeper, which makes whichever
> > > >server
> > > >that runs zookeeper as the single point of failure?
> > > >
> > > >Jason
> > > >
> > > >On Tue, Apr 23, 2013 at 8:28 AM, Chris Curtin
> > > ><[EMAIL PROTECTED]>wrote:
> > > >
> > > >> I following these instructions to get the first 'play' cluster
> going:
> > > >>
> > > >> https://cwiki.apache.org/KAFKA/kafka-08-quick-start.html
> > > >>
> > > >> Instead of running the 3 brokers on the same machine, I ran on on
> each
> > > >> machine.
> > > >>
> > > >> Note that you will need to do a little bit of zookeeper setup to
> get a
> > > >> cluster running, I think the main one was setting the 'myid' in the
> > > >> zookeeper install directory properly on each machine (namely give
> each
> > > >> machine a unique # in that file).
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Chris
> > > >>
> > > >>
> > > >> On Mon, Apr 22, 2013 at 5:32 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > >> >wrote:
> > > >>
> > > >> > Hi Jason,
> > > >> >
> > > >> > We are in the process of updating the documentation. Hoping to
> > finish
> > > >> > it by this week. Stay tuned.
> > > >> >
> > > >> > Thanks,
> > > >> > Neha
> > > >> >
> > > >> > On Mon, Apr 22, 2013 at 2:12 PM, Jason Huang <
> [EMAIL PROTECTED]
> > >
> > > >> > wrote:
> > > >> > > Hello,
> > > >> > >
> > > >> > > We've been playing around with kafka 0.8 for a few months now
> and
> > > >> decided
> > > >> > > to install kafka on a small cluster for further testing. I tried
> > to
> > > >> > search
> > > >> > > online but couldn't find any setup documentation for a kafka 0.8
> > > >> cluster.
> > > >> > >
> > > >> > > Does anyone know if such documents exist? If they don't exist,
> > what
> > > >> could
> > > >> > > be our best guide to start with?
> > > >> > >
> > > >> > > thanks!
> > > >> > >
> > > >> > > Jason
> > > >> >
> > > >>
> > >
> > >
> >
>