Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> produce request wire format question


+
Dave Peterson 2013-05-21, 16:55
+
Neha Narkhede 2013-05-21, 17:47
+
Dave Peterson 2013-05-21, 18:48
+
Colin Blower 2013-05-21, 19:05
+
Neha Narkhede 2013-05-21, 19:31
+
Dave Peterson 2013-05-21, 20:13
Copy link to this message
-
Re: produce request wire format question
Dave,

Currently, the broker expects each producer request to specify the exact
partition id (-1 is on longer valid). The mapping from a message to a
partition is done at the producer client. The producer can choose a random
partition (from the existing list of partitions) or deterministically
choose a partition based on the key.

Thanks,

Jun
On Tue, May 21, 2013 at 1:12 PM, Dave Peterson <[EMAIL PROTECTED]>wrote:

> In my case, there is a load balancer between the producers and the
> brokers, so I want the behavior described for the Java client (null key
> specifies "any partition").  If the Key field of each individual message
> specifies the partition to send it to, then I don't understand the purpose
> of the 32-bit partition identifier that precedes each message set in a
> produce request: what if a produce request specifies "partition N" for a
> given message set, and then each individual message in the set
> specifies a different partition in its Key field?  Also, the above-
> mentioned partition identifier is a 32-bit integer and the Key field of
> each individual message can contain data of arbitrary length, which
> seems inconsistent.  Is a partition identifier a 32-bit integer, or can it
> be of arbitrary length?
>
> Thanks,
> Dave
>
> On Tue, May 21, 2013 at 12:30 PM, Neha Narkhede <[EMAIL PROTECTED]>
> wrote:
> > Dave,
> >
> > Colin described the producer behavior of picking the partition for a
> > message before it is sent to Kafka broker correctly. However, I'm
> > interested in knowing your use case a little before to see why you would
> > rather have the broker decide the partition?
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, May 21, 2013 at 12:05 PM, Colin Blower <[EMAIL PROTECTED]
> >wrote:
> >
> >> The key is used by the client to decide which partition to send the
> >> message to. By the time the client is creating the produce request, it
> >> should be known which partition each message is being sent to. I believe
> >> Neha described the behavior of the Java client which sends messages with
> >> a null key to any partition.
> >>
> >> The key is described in past tense because of the use case for
> >> persisting keys with messages. The key is persisted through the broker
> >> so that a consumer knows what key was used to partition the message on
> >> the producer side.
> >>
> >> I don't believe that you can have the broker decide which partition a
> >> message goes to.
> >>
> >> --
> >> Colin B.
> >>
> >> On 05/21/2013 11:48 AM, Dave Peterson wrote:
> >> > I'm looking at the document entitled "A Guide to the Kafka Protocol"
> >> > located here:
> >> >
> >> >     https://cwiki.apache.org/KAFKA/a-guide-to-the-kafka-protocol.html
> >> >
> >> > It shows a produce request as containing a number of message sets,
> which
> >> are
> >> > grouped first by topic and second by partition (a 32-bit integer).
> >> > However, each
> >> > message in a message set contains a Key field, which is described as
> >> follows:
> >> >
> >> >     The key is an optional message key that was used for partition
> >> assignment.
> >> >     The key can be null.
> >> >
> >> > I notice the use of "was" (past tense) above.  That seems to suggest
> >> that the
> >> > Key field was once used to specify a partition (at the granularity of
> >> each
> >> > individual message), but the plan for the future is to instead use the
> >> 32-bit
> >> > partition value preceding each message set.  Is this correct?  If so,
> >> when I am
> >> > creating a produce request for 0.8, what should I use for the 32-bit
> >> partition
> >> > value, and how does this relate to the Key field of each individual
> >> message?
> >> > Ideally, I would like to just send a produce request and let the
> broker
> >> choose
> >> > the partition.  How do I accomplish this in 0.8, and are there plans
> to
> >> change
> >> > this after 0.8?
> >> >
> >> > Thanks,
> >> > Dave
> >> >
> >> > On Tue, May 21, 2013 at 10:47 AM, Neha Narkhede <
> [EMAIL PROTECTED]>

 
+
Dave Peterson 2013-05-22, 20:33
+
Neha Narkhede 2013-05-22, 23:30
+
Dave Peterson 2013-05-23, 16:43
+
Neha Narkhede 2013-05-23, 16:56
+
Colin Blower 2013-05-23, 16:57
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB