Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Non-blocking Kafka stream iterators


Copy link to this message
-
Re: Non-blocking Kafka stream iterators
Evan Chan 2013-01-22, 17:38
Jay,

For the consumer:
- Separation of the consumer logic from the main logic
- Making it easier to build the consumer for different versions of Scala
(say 2.10)
- Make it easier to read from any offset you want, while being able to keep
partition management features
- Better support for Akka and other non-blocking / event-based frameworks
(instead of a timeout, implement true hasNext functionality, for example)

thanks,
Evan
On Mon, Jan 21, 2013 at 9:27 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> It's worth mentioning that we are interested in exploring potential
> generalizations of the producer and consumer API, but as a practical matter
> most of the committers are working on getting a stable 0.8 release out the
> door. So an improved consumer and producer api would be a 0.9 feature.
>
> If you have a concrete thing you are trying to do now that is awkward it
> would be great to hear about the use case.
>
> Possible goals of improving the apis and client impls would include the
> following:
>
> Producer:
> 1. Include the offset in the information returned to the producer
> 2. Pipeline producer requests to improve throughput for synchronous
> production
>
> Consumer
> 1. Simplify api while supporting various advanced use cases like
> multi-stream consumption
> 2. Make partition assignment optional and server-side (this is currently
> the difference between the zk consumer and the simple consumer)
> 3. Make offset management optional
> 4. Remove threading from the consumer
> 5. Simplify consumer memory management
>
> -Jay
>
>
>
>
> On Mon, Jan 21, 2013 at 8:05 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > No, but you can implement it in your application.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Jan 20, 2013 at 11:02 PM, navneet sharma <
> > [EMAIL PROTECTED]> wrote:
> >
> > > Is there any property to make consumer work for lets say only 10 mins
> (ie
> > > some kind of timer)
> > > So, i want to close the consumer after 10 mins reading from broker..
> > >
> > > Thanks,
> > > Navneet Sharma
> > >
> > >
> > > On Mon, Jan 21, 2013 at 11:48 AM, Ryan LeCompte <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Perfect. Thanks Jun!
> > > >
> > > > Ryan
> > > >
> > > >
> > > > On Sun, Jan 20, 2013 at 10:16 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Yes, see consumer.timeout.ms in
> > > > http://kafka.apache.org/configuration.html
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Sun, Jan 20, 2013 at 7:14 PM, Ryan LeCompte <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > Does Kafka still only support blocking stream iterstors? It would
> > be
> > > > > great
> > > > > > to pass a timeout or have a poll() operation for fetching items.
> > > Right
> > > > > now
> > > > > > I'm always blocking in this call:
> > > > > > for (m <- stream) ...
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Ryan
> > > > >
> > > >
> > >
> >
>

--
--
*Evan Chan*
Senior Software Engineer |
[EMAIL PROTECTED] | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>