Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka 0.8 cluster setup?


+
Jason Huang 2013-04-22, 21:13
+
Neha Narkhede 2013-04-22, 21:32
+
Chris Curtin 2013-04-23, 12:29
+
Jason Huang 2013-04-23, 13:51
+
Eric Sites 2013-04-23, 14:22
+
Jason Huang 2013-04-23, 14:31
Copy link to this message
-
Re: Kafka 0.8 cluster setup?
Beat me to it ;)

Only caveat is I wouldn't use /tmp for this, since if you're running
tmpwatch the 'myid' file will get removed unexpectedly since it doesn't
seem to be changed at it. That was fun to find :)

We use /var/zookeeper for our storage.

Thanks,

Chris
On Tue, Apr 23, 2013 at 10:30 AM, Jason Huang <[EMAIL PROTECTED]> wrote:

> Thanks Eric - this helps quite a bit.
>
> I will play around with it.
>
> Jason
>
> On Tue, Apr 23, 2013 at 10:21 AM, Eric Sites <[EMAIL PROTECTED]
> >wrote:
>
> > Jason,
> >
> > You need to modify the ZooKeeper config and add the following:
> >
> > dataDir=/tmp/zookeeper
> >
> > initLimit=50
> > syncLimit=2
> >
> > server.1=kafka001.domain.com:2888:3888
> > server.2=kafka002.domain.com:2888:3888
> >
> > server.3=kafka003.domain.com:2888:3888
> >
> >
> >
> > # Make sure you open those 2 points on each of the servers for your
> > firewall
> >
> > The .1 after the server is the id of the zookeeper which should be stored
> > in a file
> > Named /tmp/zookeeper/myid
> >
> > Echo "1" > /tmp/zookeeper/myid`
> >
> >
> > Cheers,
> > Eric Sites
> >
> >
> > On 4/23/13 9:50 AM, "Jason Huang" <[EMAIL PROTECTED]> wrote:
> >
> > >Thanks Chris and Neha.
> > >
> > >Chris - I've been through the link you mentioned before. However, that
> > >appears to be using one instance of zookeeper, which makes whichever
> > >server
> > >that runs zookeeper as the single point of failure?
> > >
> > >Jason
> > >
> > >On Tue, Apr 23, 2013 at 8:28 AM, Chris Curtin
> > ><[EMAIL PROTECTED]>wrote:
> > >
> > >> I following these instructions to get the first 'play' cluster going:
> > >>
> > >> https://cwiki.apache.org/KAFKA/kafka-08-quick-start.html
> > >>
> > >> Instead of running the 3 brokers on the same machine, I ran on on each
> > >> machine.
> > >>
> > >> Note that you will need to do a little bit of zookeeper setup to get a
> > >> cluster running, I think the main one was setting the 'myid' in the
> > >> zookeeper install directory properly on each machine (namely give each
> > >> machine a unique # in that file).
> > >>
> > >> Thanks,
> > >>
> > >> Chris
> > >>
> > >>
> > >> On Mon, Apr 22, 2013 at 5:32 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > >> >wrote:
> > >>
> > >> > Hi Jason,
> > >> >
> > >> > We are in the process of updating the documentation. Hoping to
> finish
> > >> > it by this week. Stay tuned.
> > >> >
> > >> > Thanks,
> > >> > Neha
> > >> >
> > >> > On Mon, Apr 22, 2013 at 2:12 PM, Jason Huang <[EMAIL PROTECTED]
> >
> > >> > wrote:
> > >> > > Hello,
> > >> > >
> > >> > > We've been playing around with kafka 0.8 for a few months now and
> > >> decided
> > >> > > to install kafka on a small cluster for further testing. I tried
> to
> > >> > search
> > >> > > online but couldn't find any setup documentation for a kafka 0.8
> > >> cluster.
> > >> > >
> > >> > > Does anyone know if such documents exist? If they don't exist,
> what
> > >> could
> > >> > > be our best guide to start with?
> > >> > >
> > >> > > thanks!
> > >> > >
> > >> > > Jason
> > >> >
> > >>
> >
> >
>

 
+
Jason Huang 2013-04-23, 14:49
+
Jun Rao 2013-04-23, 15:43
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB