Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: is it possible to commit offsets on a per stream basis?


Copy link to this message
-
Re: is it possible to commit offsets on a per stream basis?
1 We can create multiple connectors. From each connector create only one
stream.
2 Use a single thread for a stream. In this case, the connector in each
thread can commit
freely without any dependence on the other threads.  Is this the right way
to go? Will it
introduce any dead lock when multiple connectors commit at the same time?

This is a better approach as there is no complex locking involved.

Thanks,
Neha
On Thu, Aug 29, 2013 at 10:28 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:

> Hi team,
>
> This is our current use case:
> Assume there is a topic with multiple partitions.
> 1 Create a connector first and create multiple streams from the connector
> for a topic.
> 2 Create multiple threads, one for each stream. You can assume the
> thread's job is to
> save the message into the database.
> 3 When it is time to commit offsets, all threads have to synchronize on a
> barrier
> before committing the offsets. This is to ensure no message loss in case
> of process
> crash.
>
> As all threads need to synchronize before committing, it is not efficient.
> This is a workaround:
>
> 1 We can create multiple connectors. From each connector create only one
> stream.
> 2 Use a single thread for a stream. In this case, the connector in each
> thread can commit
> freely without any dependence on the other threads.  Is this the right way
> to go? Will it
> introduce any dead lock when multiple connectors commit at the same time?
>
> It would be great to allow committing on a per stream basis.
>
> Regards,
>
> Libo
>
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB