Dave Peterson 2013-05-21, 16:55
Neha Narkhede 2013-05-21, 17:47
Dave Peterson 2013-05-21, 18:48
Colin Blower 2013-05-21, 19:05
Neha Narkhede 2013-05-21, 19:31
Dave Peterson 2013-05-21, 20:13
-Re: produce request wire format question
Jun Rao 2013-05-22, 16:28
Currently, the broker expects each producer request to specify the exact
partition id (-1 is on longer valid). The mapping from a message to a
partition is done at the producer client. The producer can choose a random
partition (from the existing list of partitions) or deterministically
choose a partition based on the key.
On Tue, May 21, 2013 at 1:12 PM, Dave Peterson <[EMAIL PROTECTED]>wrote:
> In my case, there is a load balancer between the producers and the
> brokers, so I want the behavior described for the Java client (null key
> specifies "any partition"). If the Key field of each individual message
> specifies the partition to send it to, then I don't understand the purpose
> of the 32-bit partition identifier that precedes each message set in a
> produce request: what if a produce request specifies "partition N" for a
> given message set, and then each individual message in the set
> specifies a different partition in its Key field? Also, the above-
> mentioned partition identifier is a 32-bit integer and the Key field of
> each individual message can contain data of arbitrary length, which
> seems inconsistent. Is a partition identifier a 32-bit integer, or can it
> be of arbitrary length?
> On Tue, May 21, 2013 at 12:30 PM, Neha Narkhede <[EMAIL PROTECTED]>
> > Dave,
> > Colin described the producer behavior of picking the partition for a
> > message before it is sent to Kafka broker correctly. However, I'm
> > interested in knowing your use case a little before to see why you would
> > rather have the broker decide the partition?
> > Thanks,
> > Neha
> > On Tue, May 21, 2013 at 12:05 PM, Colin Blower <[EMAIL PROTECTED]
> >> The key is used by the client to decide which partition to send the
> >> message to. By the time the client is creating the produce request, it
> >> should be known which partition each message is being sent to. I believe
> >> Neha described the behavior of the Java client which sends messages with
> >> a null key to any partition.
> >> The key is described in past tense because of the use case for
> >> persisting keys with messages. The key is persisted through the broker
> >> so that a consumer knows what key was used to partition the message on
> >> the producer side.
> >> I don't believe that you can have the broker decide which partition a
> >> message goes to.
> >> --
> >> Colin B.
> >> On 05/21/2013 11:48 AM, Dave Peterson wrote:
> >> > I'm looking at the document entitled "A Guide to the Kafka Protocol"
> >> > located here:
> >> >
> >> > https://cwiki.apache.org/KAFKA/a-guide-to-the-kafka-protocol.html
> >> >
> >> > It shows a produce request as containing a number of message sets,
> >> are
> >> > grouped first by topic and second by partition (a 32-bit integer).
> >> > However, each
> >> > message in a message set contains a Key field, which is described as
> >> follows:
> >> >
> >> > The key is an optional message key that was used for partition
> >> assignment.
> >> > The key can be null.
> >> >
> >> > I notice the use of "was" (past tense) above. That seems to suggest
> >> that the
> >> > Key field was once used to specify a partition (at the granularity of
> >> each
> >> > individual message), but the plan for the future is to instead use the
> >> 32-bit
> >> > partition value preceding each message set. Is this correct? If so,
> >> when I am
> >> > creating a produce request for 0.8, what should I use for the 32-bit
> >> partition
> >> > value, and how does this relate to the Key field of each individual
> >> message?
> >> > Ideally, I would like to just send a produce request and let the
> >> choose
> >> > the partition. How do I accomplish this in 0.8, and are there plans
> >> change
> >> > this after 0.8?
> >> >
> >> > Thanks,
> >> > Dave
> >> >
> >> > On Tue, May 21, 2013 at 10:47 AM, Neha Narkhede <
> [EMAIL PROTECTED]>
Dave Peterson 2013-05-22, 20:33
Neha Narkhede 2013-05-22, 23:30
Dave Peterson 2013-05-23, 16:43
Neha Narkhede 2013-05-23, 16:56
Colin Blower 2013-05-23, 16:57