Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> New Producer Public API


Copy link to this message
-
Re: New Producer Public API
Is the new producer API going to maintain protocol compatibility with old version if the API under the hood?

> On Jan 29, 2014, at 10:15, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
> The challenge of directly exposing ProduceRequestResult is that the offset
> provided is just the base offset and there is no way to know for a
> particular message where it was in relation to that base offset because the
> batching is transparent and non-deterministic. So I think we do need some
> kind of per-message result.
>
> I started with Future<RequestResult>, I think for the same reason you
> prefer it but then when I actually looked at some code samples it wasn't
> too great--checked exceptions, methods that we can't easily implement, etc.
> I moved away from that for two reasons:
> 1. When I actually wrote out some code samples of usage they were a little
> ugly for the reasons I described--checked exceptions, methods we can't
> implement, no helper methods, etc.
> 2. I originally intended to make the result send work like a
> ListenableFuture so that you would register the callback on the result
> rather than as part of the call. I moved away from this primarily because
> the implementation complexity was a little higher.
>
> Whether or not the code prettiness on its own outweighs the familiarity of
> a normal Future I don't know, but that was the evolution of my thinking.
>
> -Jay
>
>
>> On Wed, Jan 29, 2014 at 10:06 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>>
>> Hey Neha,
>>
>> Error handling in RecordSend works as in Future you will get the exception
>> if there is one from any of the accessor methods or await().
>>
>> The purpose of hasError was that you can write things slightly more simply
>> (which some people expressed preference for):
>>  if(send.hasError())
>>    // do something
>>  long offset = send.offset();
>>
>> Instead of the more the slightly longer:
>> try {
>>   long offset = send.offset();
>> } catch (KafkaException e) {
>>   // do something
>> }
>>
>>
>> On Wed, Jan 29, 2014 at 10:01 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>>
>>> Regarding the use of Futures -
>>>
>>> Agree that there are some downsides to using Futures but both approaches
>>> have some tradeoffs.
>>>
>>> - Standardization and usability
>>> Future is a widely used and understood Java API and given that the
>>> functionality that RecordSend hopes to provide is essentially that of
>>> Future, I think it makes sense to expose a widely understood public API
>>> for
>>> our clients. RecordSend, on the other hand, seems to provide some APIs
>>> that
>>> are very similar to that of Future, in addition to exposing a bunch of
>>> APIs
>>> that belong to ProduceRequestResult. As a user, I would've really
>>> preferred
>>> to deal with ProduceRequestResult directly -
>>> Future<ProduceRequestResult> send(...)
>>>
>>> - Error handling
>>> RecordSend's error handling is quite unintuitive where the user has to
>>> remember to invoke hasError and error, instead of just throwing the
>>> exception. Now there are
>>> some downsides regarding error handling with the Future as well, where the
>>> user has to catch InterruptedException when we would never run into it.
>>> However, it seems like a price worth paying for supporting a standard API
>>> and error handling
>>>
>>> - Unused APIs
>>> This is a downside of using Future, where the cancel() operation would
>>> always return false and mean nothing. But we can mention that caveat in
>>> our
>>> Java docs.
>>>
>>> To summarize, I would prefer to expose a well understood and widely
>>> adopted
>>> Java API and put up with the overhead of catching one unnecessary checked
>>> exception, rather than wrap the useful ProduceRequestResult in a custom
>>> async object (RecordSend) and explain that to our many users.
>>>
>>> Thanks,
>>> Neha
>>>
>>>
>>>
>>>
>>>> On Tue, Jan 28, 2014 at 8:10 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>>>>
>>>> Hey Neha,
>>>>
>>>> Can you elaborate on why you prefer using Java's Future? The downside