Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Re: Is it possible to get latest offset from kafka server?


Copy link to this message
-
Re: Is it possible to get latest offset from kafka server?
Jun Rao 2013-07-01, 03:52
If you use MirrorMaker to mirror the data from the primary cluster to the
backup cluster, currently the offsets are not preserved btw the two
clusters. In other words, the offsets for the same message in the two
clusters are not necessarily the same.

Thanks,

Jun
On Sat, Jun 29, 2013 at 10:56 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I know kafka.tools.ConsumerOffsetChecker can find out the latest offset for
> a topic from log files internally. Is there any public API that users can
> use for that?
>
> This would be useful for the failover scenario. When one cluster is down,
> the
> publisher will publish to the backup cluster. But the backup cluster
> always lags
> behind. If it is possible for publisher to find out the latest offset from
> the backup,
> it can always publish from that offset assuming publisher has a large
> enough
> buffer.  This guarantees no message loss. Thanks.
>
>
> Libo
>
>