Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> 0.8 producer -- many questions


+
ben fleis 2012-11-23, 15:51
+
Jun Rao 2012-11-26, 05:30
Copy link to this message
-
Re: 0.8 producer -- many questions
Is there any 0.8 documentation available yet for all this?

On Sun, Nov 25, 2012 at 9:30 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Ben,
>
> Please see the inlined answers below.
>
> Thanks,
>
> Jun
>
> On Fri, Nov 23, 2012 at 7:51 AM, ben fleis <[EMAIL PROTECTED]> wrote:
>
> > Hello all,
> >
> > I am writing a producer client for the v0.8 protocol.  I have the wire
> > format encoding working, and much of the functionality works well.
> >  However, I have made too many assumptions along the way, and it's time
> > that I figure out the answers to these things.  Sorry for the 'laundry
> > list' -- most of them are related, and can possibly be answered in batch
> > form.
> >
> > TopicMetadata updates
> >
> >    - is there a particular expectation of polling frequency?  (per topic,
> >    per broker, etc.?)
> >
> You just need to do the update the first time you send a message to a
> topic/partition or you get an error or an exception while sending a message
> (likely the leader has moved to a different broker). In general, you should
> be updating the metadata infrequently.
>
>
> >    - should a particular topic's metadata request only go to the
> "current"
> >    leader of that topic?
> >
>
> Metadata request can be answered by any broker.
>
>
> >    - if such a request fails to receive response, is there a particular
> >    expected behavior?
> >
>
> Then you send the same metadata request to another broker. Also, if the
> leader in the metadata response is null, it means that the leader is not
> available yet. You will need to wait a bit and retry the metadata request.
>
>
> >    - I am assuming that producers always push to the leader (feels like
> >    it's in the docs someplace, but can't confirm)
> >
>
> That's right. Producer/Fetch request can only be served at the leader of a
> partition.
>
>
> >    - if the leader changes, what is the protocol for a producer?  must
> the
> >    producer await ACK from outstanding requests before making a change,
> > and if
> >    so, presumeably re-push those requests?
> >
> >
> If the leader changes, current producer will receive an error code or an
> exception. This means that the produced data is not guaranteed to be in the
> broker. Depending on the application, the producer may choose to re-send.
>
>
> > I'm sure there will be more, but this opens the door.
> >
> > Cheers,
> >
> > ben
> >
>
+
ben fleis 2012-11-26, 11:20
+
Joel Koshy 2012-11-27, 00:32
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB