Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> kafka-49


That would seem sub-optimal.. Perhaps you could have a separate vip for producing that was configured by some process watching zk to determine which is leader or some custom health check maybe via restful api..just a thought.

Sent from my iPhone

On Jul 21, 2011, at 6:33 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> If we have ACK, the producer can catch any exception and resend.
>
> Jun
>
> On Thu, Jul 21, 2011 at 12:47 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
>> Yeah, that makes sense. How are we going to handle production through a vip
>> in that case?
>>
>> -Jay
>>
>> On Thu, Jul 21, 2011 at 7:44 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>>
>>> Without replication, we can make ACK optional. With replication, a
>> producer
>>> can only write to the leader replica. Without ACK, there is no way that
>> the
>>> broker can inform the producer that it's trying to write to the wrong
>>> broker.
>>>
>>> Jun
>>>
>>> On Wed, Jul 20, 2011 at 8:58 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>>>
>>>> It might be nice to consider making the ack optional and part of the
>>>> request. The current behavior is good for many uses, the request is
>>>> instantaneously written to the socket buffer but sent asynchronously. I
>>>> think that is a valuable use case where you care about throughput. I
>>> guess
>>>> the question is whether the asynchronous api already covers that well
>>>> enough
>>>> and how much complexity exposing that causes.
>>>>
>>>> -Jay
>>>>
>>>> On Wed, Jul 20, 2011 at 5:51 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Jeff,
>>>>>
>>>>> I was thinking of making the ACK mandatory for the producer. The ACK
>>> can
>>>> be
>>>>> sent when the message either hits 1 replicas or multiple replicas,
>>>>> depending
>>>>> on the setting.
>>>>>
>>>>> Having the ACK include the starting offset of the message seems
>>>> reasonable.
>>>>> It will be a bit complicated for multisend since multiple offsets
>> have
>>> to
>>>>> be
>>>>> returned. What do you need the offset for?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>> On Wed, Jul 20, 2011 at 12:47 PM, Jeffrey Damick <
>>>> [EMAIL PROTECTED]
>>>>>> wrote:
>>>>>
>>>>>> Is there any current thought around KAFKA-49, for acknowledgement
>> of
>>>>>> producers?
>>>>>> Will this be optional, a new message type(s)?
>>>>>> Will the ack be synchronous or asynchronous or depending on request
>>>> type?
>>>>>>
>>>>>> It would be fantastic if the ack contained the starting offset of
>> the
>>>>>> message published, and not just the ending.
>>>>>>
>>>>>> This is quickly becoming an issue for us, so we may be able to
>>> provide
>>>>> some
>>>>>> help in this area..
>>>>>>
>>>>>>
>>>>>> thanks
>>>>>> -jeff
>>>>>>
>>>>>
>>>>
>>>
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB