Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka wiki Documentation conventions - looking for feedback


+
Chris Curtin 2013-03-25, 12:03
+
Chris Curtin 2013-03-29, 15:28
+
Jun Rao 2013-04-22, 16:27
+
Chris Curtin 2013-04-22, 17:41
+
Jun Rao 2013-04-22, 22:12
+
Jun Rao 2013-04-29, 04:51
+
Chris Curtin 2013-04-29, 12:55
+
Jun Rao 2013-04-29, 15:03
+
Chris Curtin 2013-04-29, 15:17
+
Jun Rao 2013-04-29, 15:58
+
Chris Curtin 2013-04-29, 18:19
+
Jun Rao 2013-04-29, 22:18
Copy link to this message
-
Re: Kafka wiki Documentation conventions - looking for feedback
I've tested my examples with the new (4/30) release and they work, so I've
updated the documentation.

Thanks,

Chris
On Mon, Apr 29, 2013 at 6:18 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Thanks. I also updated your producer example to reflect a recent config
> change (broker.list => metadata.broker.list).
>
> Jun
>
>
> On Mon, Apr 29, 2013 at 11:19 AM, Chris Curtin <[EMAIL PROTECTED]
> >wrote:
>
> > Thanks, I missed that the addition of consumers can cause a re-balance.
> > Thought it was only on Leader changes.
> >
> > I've updated the wording in the example.
> >
> > I'll pull down the beta and test my application then change the names on
> > the properties.
> >
> > Thanks,
> >
> > Chris
> >
> >
> > On Mon, Apr 29, 2013 at 11:58 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Basically, every time a consumer joins a group, every consumer in the
> > > groups gets a ZK notification and each of them tries to own a subset of
> > the
> > > total number of partitions. A given partition is only assigned to one
> of
> > > the consumers in the same group. Once the ownership is determined, each
> > > consumer consumes messages coming from its partitions and manages the
> > > offset of those partitions. Since at any given point of time, a
> partition
> > > is only owned by one consumer, there won't be conflicts on updating the
> > > offsets. More details are described in the "consumer rebalancing
> > algorithm"
> > > section of http://kafka.apache.org/07/design.html
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Apr 29, 2013 at 8:16 AM, Chris Curtin <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Jun, can you explain this a little better? I thought when using
> > Consumer
> > > > Groups that on startup Kafka connects to ZooKeeper and finds the last
> > > read
> > > > offset for every partition in the topic being requested for the
> group.
> > > That
> > > > is then the starting point for the consumer threads.
> > > >
> > > > If a second process starts while the first one is running with the
> same
> > > > Consumer Group, won't the second one read the last offsets consumed
> by
> > > the
> > > > already running process and start processing from there? Then as the
> > > first
> > > > process syncs consumed offsets, won't the 2nd process's next update
> > > > overwrite them?
> > > >
> > > > Thanks,
> > > >
> > > > Chris
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Apr 29, 2013 at 11:03 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Chris,
> > > > >
> > > > > Thanks for the writeup. Looks great overall. A couple of comments.
> > > > >
> > > > > 1. At the beginning, it sounds like that one can't run multiple
> > > processes
> > > > > of consumers in the same group. This is actually not true. We can
> > > create
> > > > > multiple instances of consumers for the same group in the same JVM
> or
> > > > > different JVMs. The consumers will auto-balance among themselves.
> > > > >
> > > > > 2. We have changed the name of some config properties.
> > > > > auto.commit.interval.ms is correct. However, zk.connect,
> > > > > zk.session.timeout.ms and zk.sync.time.ms are changed to
> > > > > zookeeper.connect,
> > > > > zookeeper.session.timeout.ms, and zookeeper.sync.time.ms,
> > > respectively.
> > > > >
> > > > > I will add a link to your wiki in our website.
> > > > >
> > > > > Thanks again.
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Mon, Apr 29, 2013 at 5:54 AM, Chris Curtin <
> > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > I finished and published it this morning:
> > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > > > >
> > > > > > One question: when documenting the ConsumerConfig parameters I
> > > couldn't
> > > > > > find a description for the 'auto.commit.interval.ms' setting. I
> > > found
> > > > > one
> > > > > > for 'autocommit.interval.ms' (no '.' between auto and commit) in

 
+
Jun Rao 2013-05-01, 15:46
+
Chris Curtin 2013-05-01, 19:34
+
Jun Rao 2013-05-01, 21:37
+
Jun Rao 2013-04-24, 16:26
+
Chris Curtin 2013-04-24, 18:12
+
Jun Rao 2013-04-25, 04:08
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB