Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - produce request wire format question


Copy link to this message
-
Re: produce request wire format question
Neha Narkhede 2013-05-21, 19:31
Dave,

Colin described the producer behavior of picking the partition for a
message before it is sent to Kafka broker correctly. However, I'm
interested in knowing your use case a little before to see why you would
rather have the broker decide the partition?

Thanks,
Neha
On Tue, May 21, 2013 at 12:05 PM, Colin Blower <[EMAIL PROTECTED]>wrote:

> The key is used by the client to decide which partition to send the
> message to. By the time the client is creating the produce request, it
> should be known which partition each message is being sent to. I believe
> Neha described the behavior of the Java client which sends messages with
> a null key to any partition.
>
> The key is described in past tense because of the use case for
> persisting keys with messages. The key is persisted through the broker
> so that a consumer knows what key was used to partition the message on
> the producer side.
>
> I don't believe that you can have the broker decide which partition a
> message goes to.
>
> --
> Colin B.
>
> On 05/21/2013 11:48 AM, Dave Peterson wrote:
> > I'm looking at the document entitled "A Guide to the Kafka Protocol"
> > located here:
> >
> >     https://cwiki.apache.org/KAFKA/a-guide-to-the-kafka-protocol.html
> >
> > It shows a produce request as containing a number of message sets, which
> are
> > grouped first by topic and second by partition (a 32-bit integer).
> > However, each
> > message in a message set contains a Key field, which is described as
> follows:
> >
> >     The key is an optional message key that was used for partition
> assignment.
> >     The key can be null.
> >
> > I notice the use of "was" (past tense) above.  That seems to suggest
> that the
> > Key field was once used to specify a partition (at the granularity of
> each
> > individual message), but the plan for the future is to instead use the
> 32-bit
> > partition value preceding each message set.  Is this correct?  If so,
> when I am
> > creating a produce request for 0.8, what should I use for the 32-bit
> partition
> > value, and how does this relate to the Key field of each individual
> message?
> > Ideally, I would like to just send a produce request and let the broker
> choose
> > the partition.  How do I accomplish this in 0.8, and are there plans to
> change
> > this after 0.8?
> >
> > Thanks,
> > Dave
> >
> > On Tue, May 21, 2013 at 10:47 AM, Neha Narkhede <[EMAIL PROTECTED]>
> wrote:
> >> No. In 0.8, if you don't specify a key for a message, it is sent to any
> of
> >> the available partitions. In other words, the partition id is selected
> on
> >> the partition and the server doesn't get -1 as the partition id.
> >>
> >> Thanks,
> >> Neha
> >>
> >>
> >> On Tue, May 21, 2013 at 9:54 AM, Dave Peterson <[EMAIL PROTECTED]
> >wrote:
> >>
> >>> In the version 0.8 wire format for a produce request, does a value of
> -1
> >>> still indicate "use a random partition" as it did for 0.7?
> >>>
> >>> Thanks,
> >>> Dave
> >>>
>
>
>