Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - produce request wire format question


Copy link to this message
-
Re: produce request wire format question
Neha Narkhede 2013-05-23, 16:56
Yes. You can see the details of the wire protocol here
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-MetadataResponse

Thanks,
Neha
On May 23, 2013 9:43 AM, "Dave Peterson" <[EMAIL PROTECTED]> wrote:

> Ok, thanks for the information.  Looking at the wire format for the
> metadata response, I see that the right hand side of the TopicMetadata
> production contains a TopicErrorCode, and the right hand side of the
> PartitionMetadata production contains a PartitionErrorCode.  Are both
> of these 16-bit values?  In general, where it isn't stated explicitly in
> the documentation, can I assume that all error codes are 16-bit values?
>
> Thanks,
> Dave
>
>
> On Wed, May 22, 2013 at 4:29 PM, Neha Narkhede <[EMAIL PROTECTED]>
> wrote:
> > 1. Correct
> > 2. The producer does not use or depend on zookeeper anymore. It refreshes
> > its view of the cluster metadata by using a TopicMetadataRequest to any
> of
> > the kafka brokers. It maps a message to a partition using the following
> > rules -
> > 2.1 If a message has no key, use any available partition
> > 2.2 If a message has a key and the user has defined a custom partitioner,
> > use it to map the key to a partition id
> > 2.3 If a message has a key and the user has not defined a custom
> > partitioner, use the default hash based partitioner that ships with Kafka
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, May 22, 2013 at 1:33 PM, Dave Peterson <[EMAIL PROTECTED]
> >wrote:
> >
> >> Ok, the picture I have in my mind of how things work in 0.8 (from a
> >> producer's point of view) is as follows:
> >>
> >>     1.  An application program sends log messages to a producer.  Each
> >>         message is provided as a key/value pair, where the key is chosen
> >>         by the application and the value is the message contents.  By
> its
> >>         choice of key, the application may influence or control which
> >>         partition the message gets sent to.
> >>
> >>     2.  The producer receives messages as key/value pairs.  From talking
> >>         with zookeeper, it knows the set of available brokers and which
> >>         partitions each broker has.  If the sending application
> provided a
> >> key
> >>         for a given message, the contents of the key may optionally
> >>         influence the producer's choice of broker and partition to send
> the
> >>         message to, according to some convention understood by both
> >>         application program and producer.
> >>
> >> Is this correct?
> >>
> >> Thanks,
> >> Dave
> >>
> >> On Wed, May 22, 2013 at 9:28 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >> > Dave,
> >> >
> >> > Currently, the broker expects each producer request to specify the
> exact
> >> > partition id (-1 is on longer valid). The mapping from a message to a
> >> > partition is done at the producer client. The producer can choose a
> >> random
> >> > partition (from the existing list of partitions) or deterministically
> >> > choose a partition based on the key.
> >> >
> >> > Thanks,
> >> >
> >> > Jun
> >> >
> >> >
> >> > On Tue, May 21, 2013 at 1:12 PM, Dave Peterson <[EMAIL PROTECTED]
> >> >wrote:
> >> >
> >> >> In my case, there is a load balancer between the producers and the
> >> >> brokers, so I want the behavior described for the Java client (null
> key
> >> >> specifies "any partition").  If the Key field of each individual
> message
> >> >> specifies the partition to send it to, then I don't understand the
> >> purpose
> >> >> of the 32-bit partition identifier that precedes each message set in
> a
> >> >> produce request: what if a produce request specifies "partition N"
> for a
> >> >> given message set, and then each individual message in the set
> >> >> specifies a different partition in its Key field?  Also, the above-
> >> >> mentioned partition identifier is a 32-bit integer and the Key field
> of
> >> >> each individual message can contain data of arbitrary length, which
> >> >> seems inconsistent.  Is a partition identifier a 32-bit integer, or