Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Kafka wiki Documentation conventions - looking for feedback


+
Chris Curtin 2013-03-25, 12:03
+
Chris Curtin 2013-03-29, 15:28
+
Jun Rao 2013-04-22, 16:27
+
Chris Curtin 2013-04-22, 17:41
+
Jun Rao 2013-04-22, 22:12
+
Jun Rao 2013-04-29, 04:51
+
Chris Curtin 2013-04-29, 12:55
+
Jun Rao 2013-04-29, 15:03
+
Chris Curtin 2013-04-29, 15:17
+
Jun Rao 2013-04-29, 15:58
+
Chris Curtin 2013-04-29, 18:19
+
Jun Rao 2013-04-29, 22:18
Copy link to this message
-
Re: Kafka wiki Documentation conventions - looking for feedback
Chris Curtin 2013-05-01, 13:56
I've tested my examples with the new (4/30) release and they work, so I've
updated the documentation.

Thanks,

Chris
On Mon, Apr 29, 2013 at 6:18 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Thanks. I also updated your producer example to reflect a recent config
> change (broker.list => metadata.broker.list).
>
> Jun
>
>
> On Mon, Apr 29, 2013 at 11:19 AM, Chris Curtin <[EMAIL PROTECTED]
> >wrote:
>
> > Thanks, I missed that the addition of consumers can cause a re-balance.
> > Thought it was only on Leader changes.
> >
> > I've updated the wording in the example.
> >
> > I'll pull down the beta and test my application then change the names on
> > the properties.
> >
> > Thanks,
> >
> > Chris
> >
> >
> > On Mon, Apr 29, 2013 at 11:58 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Basically, every time a consumer joins a group, every consumer in the
> > > groups gets a ZK notification and each of them tries to own a subset of
> > the
> > > total number of partitions. A given partition is only assigned to one
> of
> > > the consumers in the same group. Once the ownership is determined, each
> > > consumer consumes messages coming from its partitions and manages the
> > > offset of those partitions. Since at any given point of time, a
> partition
> > > is only owned by one consumer, there won't be conflicts on updating the
> > > offsets. More details are described in the "consumer rebalancing
> > algorithm"
> > > section of http://kafka.apache.org/07/design.html
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Apr 29, 2013 at 8:16 AM, Chris Curtin <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Jun, can you explain this a little better? I thought when using
> > Consumer
> > > > Groups that on startup Kafka connects to ZooKeeper and finds the last
> > > read
> > > > offset for every partition in the topic being requested for the
> group.
> > > That
> > > > is then the starting point for the consumer threads.
> > > >
> > > > If a second process starts while the first one is running with the
> same
> > > > Consumer Group, won't the second one read the last offsets consumed
> by
> > > the
> > > > already running process and start processing from there? Then as the
> > > first
> > > > process syncs consumed offsets, won't the 2nd process's next update
> > > > overwrite them?
> > > >
> > > > Thanks,
> > > >
> > > > Chris
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Apr 29, 2013 at 11:03 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Chris,
> > > > >
> > > > > Thanks for the writeup. Looks great overall. A couple of comments.
> > > > >
> > > > > 1. At the beginning, it sounds like that one can't run multiple
> > > processes
> > > > > of consumers in the same group. This is actually not true. We can
> > > create
> > > > > multiple instances of consumers for the same group in the same JVM
> or
> > > > > different JVMs. The consumers will auto-balance among themselves.
> > > > >
> > > > > 2. We have changed the name of some config properties.
> > > > > auto.commit.interval.ms is correct. However, zk.connect,
> > > > > zk.session.timeout.ms and zk.sync.time.ms are changed to
> > > > > zookeeper.connect,
> > > > > zookeeper.session.timeout.ms, and zookeeper.sync.time.ms,
> > > respectively.
> > > > >
> > > > > I will add a link to your wiki in our website.
> > > > >
> > > > > Thanks again.
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Mon, Apr 29, 2013 at 5:54 AM, Chris Curtin <
> > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > I finished and published it this morning:
> > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> > > > > >
> > > > > > One question: when documenting the ConsumerConfig parameters I
> > > couldn't
> > > > > > find a description for the 'auto.commit.interval.ms' setting. I
> > > found
> > > > > one
> > > > > > for 'autocommit.interval.ms' (no '.' between auto and commit) in

 
+
Jun Rao 2013-05-01, 15:46
+
Chris Curtin 2013-05-01, 19:34
+
Jun Rao 2013-05-01, 21:37
+
Jun Rao 2013-04-24, 16:26
+
Chris Curtin 2013-04-24, 18:12
+
Jun Rao 2013-04-25, 04:08