Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> 0.8 producer -- many questions


Copy link to this message
-
Re: 0.8 producer -- many questions
Is there any 0.8 documentation available yet for all this?

On Sun, Nov 25, 2012 at 9:30 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Ben,
>
> Please see the inlined answers below.
>
> Thanks,
>
> Jun
>
> On Fri, Nov 23, 2012 at 7:51 AM, ben fleis <[EMAIL PROTECTED]> wrote:
>
> > Hello all,
> >
> > I am writing a producer client for the v0.8 protocol.  I have the wire
> > format encoding working, and much of the functionality works well.
> >  However, I have made too many assumptions along the way, and it's time
> > that I figure out the answers to these things.  Sorry for the 'laundry
> > list' -- most of them are related, and can possibly be answered in batch
> > form.
> >
> > TopicMetadata updates
> >
> >    - is there a particular expectation of polling frequency?  (per topic,
> >    per broker, etc.?)
> >
> You just need to do the update the first time you send a message to a
> topic/partition or you get an error or an exception while sending a message
> (likely the leader has moved to a different broker). In general, you should
> be updating the metadata infrequently.
>
>
> >    - should a particular topic's metadata request only go to the
> "current"
> >    leader of that topic?
> >
>
> Metadata request can be answered by any broker.
>
>
> >    - if such a request fails to receive response, is there a particular
> >    expected behavior?
> >
>
> Then you send the same metadata request to another broker. Also, if the
> leader in the metadata response is null, it means that the leader is not
> available yet. You will need to wait a bit and retry the metadata request.
>
>
> >    - I am assuming that producers always push to the leader (feels like
> >    it's in the docs someplace, but can't confirm)
> >
>
> That's right. Producer/Fetch request can only be served at the leader of a
> partition.
>
>
> >    - if the leader changes, what is the protocol for a producer?  must
> the
> >    producer await ACK from outstanding requests before making a change,
> > and if
> >    so, presumeably re-push those requests?
> >
> >
> If the leader changes, current producer will receive an error code or an
> exception. This means that the produced data is not guaranteed to be in the
> broker. Depending on the application, the producer may choose to re-send.
>
>
> > I'm sure there will be more, but this opens the door.
> >
> > Cheers,
> >
> > ben
> >
>