Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Non-blocking Kafka stream iterators


Copy link to this message
-
Re: Non-blocking Kafka stream iterators
Jay,

For the consumer:
- Separation of the consumer logic from the main logic
- Making it easier to build the consumer for different versions of Scala
(say 2.10)
- Make it easier to read from any offset you want, while being able to keep
partition management features
- Better support for Akka and other non-blocking / event-based frameworks
(instead of a timeout, implement true hasNext functionality, for example)

thanks,
Evan
On Mon, Jan 21, 2013 at 9:27 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> It's worth mentioning that we are interested in exploring potential
> generalizations of the producer and consumer API, but as a practical matter
> most of the committers are working on getting a stable 0.8 release out the
> door. So an improved consumer and producer api would be a 0.9 feature.
>
> If you have a concrete thing you are trying to do now that is awkward it
> would be great to hear about the use case.
>
> Possible goals of improving the apis and client impls would include the
> following:
>
> Producer:
> 1. Include the offset in the information returned to the producer
> 2. Pipeline producer requests to improve throughput for synchronous
> production
>
> Consumer
> 1. Simplify api while supporting various advanced use cases like
> multi-stream consumption
> 2. Make partition assignment optional and server-side (this is currently
> the difference between the zk consumer and the simple consumer)
> 3. Make offset management optional
> 4. Remove threading from the consumer
> 5. Simplify consumer memory management
>
> -Jay
>
>
>
>
> On Mon, Jan 21, 2013 at 8:05 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > No, but you can implement it in your application.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Jan 20, 2013 at 11:02 PM, navneet sharma <
> > [EMAIL PROTECTED]> wrote:
> >
> > > Is there any property to make consumer work for lets say only 10 mins
> (ie
> > > some kind of timer)
> > > So, i want to close the consumer after 10 mins reading from broker..
> > >
> > > Thanks,
> > > Navneet Sharma
> > >
> > >
> > > On Mon, Jan 21, 2013 at 11:48 AM, Ryan LeCompte <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Perfect. Thanks Jun!
> > > >
> > > > Ryan
> > > >
> > > >
> > > > On Sun, Jan 20, 2013 at 10:16 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Yes, see consumer.timeout.ms in
> > > > http://kafka.apache.org/configuration.html
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Sun, Jan 20, 2013 at 7:14 PM, Ryan LeCompte <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > Does Kafka still only support blocking stream iterstors? It would
> > be
> > > > > great
> > > > > > to pass a timeout or have a poll() operation for fetching items.
> > > Right
> > > > > now
> > > > > > I'm always blocking in this call:
> > > > > > for (m <- stream) ...
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Ryan
> > > > >
> > > >
> > >
> >
>

--
--
*Evan Chan*
Senior Software Engineer |
[EMAIL PROTECTED] | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB