Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - kafka-49


Copy link to this message
-
Re: kafka-49
Jeffrey Damick 2011-07-22, 01:08
That would seem sub-optimal.. Perhaps you could have a separate vip for producing that was configured by some process watching zk to determine which is leader or some custom health check maybe via restful api..just a thought.

Sent from my iPhone

On Jul 21, 2011, at 6:33 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> If we have ACK, the producer can catch any exception and resend.
>
> Jun
>
> On Thu, Jul 21, 2011 at 12:47 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
>> Yeah, that makes sense. How are we going to handle production through a vip
>> in that case?
>>
>> -Jay
>>
>> On Thu, Jul 21, 2011 at 7:44 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>>
>>> Without replication, we can make ACK optional. With replication, a
>> producer
>>> can only write to the leader replica. Without ACK, there is no way that
>> the
>>> broker can inform the producer that it's trying to write to the wrong
>>> broker.
>>>
>>> Jun
>>>
>>> On Wed, Jul 20, 2011 at 8:58 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>>>
>>>> It might be nice to consider making the ack optional and part of the
>>>> request. The current behavior is good for many uses, the request is
>>>> instantaneously written to the socket buffer but sent asynchronously. I
>>>> think that is a valuable use case where you care about throughput. I
>>> guess
>>>> the question is whether the asynchronous api already covers that well
>>>> enough
>>>> and how much complexity exposing that causes.
>>>>
>>>> -Jay
>>>>
>>>> On Wed, Jul 20, 2011 at 5:51 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Jeff,
>>>>>
>>>>> I was thinking of making the ACK mandatory for the producer. The ACK
>>> can
>>>> be
>>>>> sent when the message either hits 1 replicas or multiple replicas,
>>>>> depending
>>>>> on the setting.
>>>>>
>>>>> Having the ACK include the starting offset of the message seems
>>>> reasonable.
>>>>> It will be a bit complicated for multisend since multiple offsets
>> have
>>> to
>>>>> be
>>>>> returned. What do you need the offset for?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>> On Wed, Jul 20, 2011 at 12:47 PM, Jeffrey Damick <
>>>> [EMAIL PROTECTED]
>>>>>> wrote:
>>>>>
>>>>>> Is there any current thought around KAFKA-49, for acknowledgement
>> of
>>>>>> producers?
>>>>>> Will this be optional, a new message type(s)?
>>>>>> Will the ack be synchronous or asynchronous or depending on request
>>>> type?
>>>>>>
>>>>>> It would be fantastic if the ack contained the starting offset of
>> the
>>>>>> message published, and not just the ending.
>>>>>>
>>>>>> This is quickly becoming an issue for us, so we may be able to
>>> provide
>>>>> some
>>>>>> help in this area..
>>>>>>
>>>>>>
>>>>>> thanks
>>>>>> -jeff
>>>>>>
>>>>>
>>>>
>>>
>>