王国栋 2013-07-03, 02:56
Jun Rao 2013-07-03, 04:54
-Re: how to keep the same partition when using mirror maker
王国栋 2013-07-03, 07:47
On Wed, Jul 3, 2013 at 12:53 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> One of the issues is that in 0.7, the partitioning key is not stored on the
> broker. So, mirror maker won't know what key to use for partitioning.
> In 0.8, the partitioning key will be stored on the broker. Once kafka-957
> is resolved, mirror maker will be able to partition messages based on the
> same key in the target cluster.
> On Tue, Jul 2, 2013 at 7:55 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
> > HI guys,
> > We are using kafka0.7.2, in our cluster, we use customized partition
> > functions in producer.
> > Say, we compute partition id with user id in our log.
> > But when we use mirror maker to pull log, we find that mirror maker uses
> > random partition to push log into destination brokers.
> > In my mind, we have to decompress logs from consumer and re-partition
> > again. So, I am wondering if there are any good ways to keep partition
> > in mirror maker.
> > Thanks,
> > Guodong