Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - produce request wire format question


Copy link to this message
-
Re: produce request wire format question
Jun Rao 2013-05-22, 16:28
Dave,

Currently, the broker expects each producer request to specify the exact
partition id (-1 is on longer valid). The mapping from a message to a
partition is done at the producer client. The producer can choose a random
partition (from the existing list of partitions) or deterministically
choose a partition based on the key.

Thanks,

Jun
On Tue, May 21, 2013 at 1:12 PM, Dave Peterson <[EMAIL PROTECTED]>wrote:

> In my case, there is a load balancer between the producers and the
> brokers, so I want the behavior described for the Java client (null key
> specifies "any partition").  If the Key field of each individual message
> specifies the partition to send it to, then I don't understand the purpose
> of the 32-bit partition identifier that precedes each message set in a
> produce request: what if a produce request specifies "partition N" for a
> given message set, and then each individual message in the set
> specifies a different partition in its Key field?  Also, the above-
> mentioned partition identifier is a 32-bit integer and the Key field of
> each individual message can contain data of arbitrary length, which
> seems inconsistent.  Is a partition identifier a 32-bit integer, or can it
> be of arbitrary length?
>
> Thanks,
> Dave
>
> On Tue, May 21, 2013 at 12:30 PM, Neha Narkhede <[EMAIL PROTECTED]>
> wrote:
> > Dave,
> >
> > Colin described the producer behavior of picking the partition for a
> > message before it is sent to Kafka broker correctly. However, I'm
> > interested in knowing your use case a little before to see why you would
> > rather have the broker decide the partition?
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, May 21, 2013 at 12:05 PM, Colin Blower <[EMAIL PROTECTED]
> >wrote:
> >
> >> The key is used by the client to decide which partition to send the
> >> message to. By the time the client is creating the produce request, it
> >> should be known which partition each message is being sent to. I believe
> >> Neha described the behavior of the Java client which sends messages with
> >> a null key to any partition.
> >>
> >> The key is described in past tense because of the use case for
> >> persisting keys with messages. The key is persisted through the broker
> >> so that a consumer knows what key was used to partition the message on
> >> the producer side.
> >>
> >> I don't believe that you can have the broker decide which partition a
> >> message goes to.
> >>
> >> --
> >> Colin B.
> >>
> >> On 05/21/2013 11:48 AM, Dave Peterson wrote:
> >> > I'm looking at the document entitled "A Guide to the Kafka Protocol"
> >> > located here:
> >> >
> >> >     https://cwiki.apache.org/KAFKA/a-guide-to-the-kafka-protocol.html
> >> >
> >> > It shows a produce request as containing a number of message sets,
> which
> >> are
> >> > grouped first by topic and second by partition (a 32-bit integer).
> >> > However, each
> >> > message in a message set contains a Key field, which is described as
> >> follows:
> >> >
> >> >     The key is an optional message key that was used for partition
> >> assignment.
> >> >     The key can be null.
> >> >
> >> > I notice the use of "was" (past tense) above.  That seems to suggest
> >> that the
> >> > Key field was once used to specify a partition (at the granularity of
> >> each
> >> > individual message), but the plan for the future is to instead use the
> >> 32-bit
> >> > partition value preceding each message set.  Is this correct?  If so,
> >> when I am
> >> > creating a produce request for 0.8, what should I use for the 32-bit
> >> partition
> >> > value, and how does this relate to the Key field of each individual
> >> message?
> >> > Ideally, I would like to just send a produce request and let the
> broker
> >> choose
> >> > the partition.  How do I accomplish this in 0.8, and are there plans
> to
> >> change
> >> > this after 0.8?
> >> >
> >> > Thanks,
> >> > Dave
> >> >
> >> > On Tue, May 21, 2013 at 10:47 AM, Neha Narkhede <
> [EMAIL PROTECTED]>