Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Java 8 influence on next-generation Java producer and consumer APIs


Copy link to this message
-
Re: Java 8 influence on next-generation Java producer and consumer APIs
Jay Kreps 2014-01-30, 18:14
I think this is a good point and you are not the first person to bring it
up.

I am not hugely knowledgable about java 8 so any feedback would be useful.

In the producer I think the biggest impact is that the Callback can be
implemented with a lambda instead of a anon class which will be much nicer.
But maybe there are other things? Would be great if someone could take a
look at that producer api with a forward looking eye...

In the consumer there is potentially a lot of similarity between Stream and
the consumer. What we had tentatively proposed in this area was actually to
replace the iterator we currently provide with a lower-level method
   List<RecordAndMetadata> poll(long timeoutMs)
This method polls over all topics and returns a list of messages for
processing. This is arguably the least convenient end-user api for some of
the simpler cases, but the advantage is that it can be implemented in a
completely single threaded fashion which allows us to full decouple the
threading model of message processing--currently the zookeeper consumer
actually requires a per-topic blocking thread. There are a number of
slightly more convenient apis that could be implemented on top of this. I
think we were just about to kick off a more detailed discussion here...

-Jay
On Wed, Jan 29, 2014 at 12:05 PM, Clark Breyman <[EMAIL PROTECTED]> wrote:

> Jay et al,
>
> What are your current thoughts on ensuring that the next-generation APIs
> play nicely with both lambdas and the extensions to the standard runtime in
> Java 8?
>
> My thoughts are that if folks are doing the work to reimplement/redesign
> the API, it should be as compatible as possible with the latest stuff
> provided it doesn't break 1.6 compatibility (as long as that's required).
>
> Areas where this might influence the design:
> - Using single-method interfaces that could be supplied by lambdas when
> running in J8
> - Consideration of Spliterator, Supplier, Stream, ... when designing their
> Kakfa counterparts.
>
> Regards,
> Clark
>