Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: is it possible to commit offsets on a per stream basis?


Copy link to this message
-
Re: is it possible to commit offsets on a per stream basis?
I added a comment/suggestion to:
https://issues.apache.org/jira/browse/KAFKA-966

Basically to expose an api for marking an offset for commit, such that the
auto-commit would only commit offsets up to the last message
'markedForCommit', and not the last 'consumed' offset, which may or may not
have succeeded.  This way, consumer code can just call 'markForCommit()'
after successfully processing each message successfully.

Does that make sense?
On Mon, Sep 9, 2013 at 5:21 PM, Yu, Libo <[EMAIL PROTECTED]> wrote:

> Thanks, Neha. That number of connections formula is very helpful.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> Sent: Monday, September 09, 2013 12:17 PM
> To: [EMAIL PROTECTED]
> Subject: Re: is it possible to commit offsets on a per stream basis?
>
> Memory might become an issue if all the connectors are part of the same
> process. But this is easily solvable by distributing the connectors over
> several machines.
> Number of connections would be (# of connectors) * (# of brokers) and will
> proportionately increase with the # of connectors.
>
> Thanks,
> Neha
>
>
> On Mon, Sep 9, 2013 at 9:08 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:
>
> > If one connector is used for a single stream, when there are many
> > topics/streams, will that cause any performance issue, e.g. too many
> > connections or too much memory or big latency?
> >
> > Regards,
> >
> > Libo
> >
> >
> > -----Original Message-----
> > From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > Sent: Sunday, September 08, 2013 12:46 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: is it possible to commit offsets on a per stream basis?
> >
> > That should be fine too.
> >
> >
> >
> >
> > On Sat, Sep 7, 2013 at 8:33 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > To be clear, it looks like I forgot to add to my question, that I am
> > > asking about creating multiple connectors, within the same consumer
> > > process (as I realize I can obviously have multiple connectors
> > > running on multiple hosts, etc.).  But I'm guessing that should be
> fine too?
> > >
> > > Jason
> > >
> > >
> > >
> > >
> > > On Sat, Sep 7, 2013 at 3:09 PM, Neha Narkhede
> > > <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > >> Can I create multiple connectors, and have each use the same
> > > > >> Regex
> > > > for the TopicFilter?  Will each connector share the set of
> > > > available topics?  Is this safe to do?
> > > >
> > > > >> Or is it necessary to create mutually non-intersecting regex's
> > > > >> for
> > > each
> > > > connector?
> > > >
> > > > As long as each of those consumer connectors share the same group
> > > > id,
> > > Kafka
> > > > consumer rebalancing should automatically re-distribute the
> > > > topic/partitions amongst the consumer connectors/streams evenly.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Mon, Sep 2, 2013 at 1:35 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Will this work if we are using a TopicFilter, that can map to
> > > > > multiple topics.  Can I create multiple connectors, and have
> > > > > each use the same
> > > > Regex
> > > > > for the TopicFilter?  Will each connector share the set of
> > > > > available topics?  Is this safe to do?
> > > > >
> > > > > Or is it necessary to create mutually non-intersecting regex's
> > > > > for each connector?
> > > > >
> > > > > It seems I have a similar issue.  I have been using auto commit
> > > > > mode,
> > > but
> > > > > it doesn't guarantee that all messages committed have been
> > > > > successfully processed (seems a change to the connector itself
> > > > > might expose a way to
> > > > use
> > > > > auto offset commit, and have it never commit a message until it
> > > > > is processed).  But that would be a change to the
> > > > > ZookeeperConsumerConnector....Essentially, it would be great if
> > > > > after processing each message, we could mark the message as

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB