Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> 0.8 producer -- many questions

ben fleis 2012-11-23, 15:51
Copy link to this message
Re: 0.8 producer -- many questions

Please see the inlined answers below.



On Fri, Nov 23, 2012 at 7:51 AM, ben fleis <[EMAIL PROTECTED]> wrote:

> Hello all,
> I am writing a producer client for the v0.8 protocol.  I have the wire
> format encoding working, and much of the functionality works well.
>  However, I have made too many assumptions along the way, and it's time
> that I figure out the answers to these things.  Sorry for the 'laundry
> list' -- most of them are related, and can possibly be answered in batch
> form.
> TopicMetadata updates
>    - is there a particular expectation of polling frequency?  (per topic,
>    per broker, etc.?)
You just need to do the update the first time you send a message to a
topic/partition or you get an error or an exception while sending a message
(likely the leader has moved to a different broker). In general, you should
be updating the metadata infrequently.
>    - should a particular topic's metadata request only go to the "current"
>    leader of that topic?

Metadata request can be answered by any broker.
>    - if such a request fails to receive response, is there a particular
>    expected behavior?

Then you send the same metadata request to another broker. Also, if the
leader in the metadata response is null, it means that the leader is not
available yet. You will need to wait a bit and retry the metadata request.
>    - I am assuming that producers always push to the leader (feels like
>    it's in the docs someplace, but can't confirm)

That's right. Producer/Fetch request can only be served at the leader of a
>    - if the leader changes, what is the protocol for a producer?  must the
>    producer await ACK from outstanding requests before making a change,
> and if
>    so, presumeably re-push those requests?
If the leader changes, current producer will receive an error code or an
exception. This means that the produced data is not guaranteed to be in the
broker. Depending on the application, the producer may choose to re-send.
> I'm sure there will be more, but this opens the door.
> Cheers,
> ben
Jason Rosenberg 2012-11-26, 06:53
ben fleis 2012-11-26, 11:20
Joel Koshy 2012-11-27, 00:32