Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> how to keep the same partition when using mirror maker


Copy link to this message
-
Re: how to keep the same partition when using mirror maker
One of the issues is that in 0.7, the partitioning key is not stored on the
broker. So, mirror maker won't know what key to use for partitioning.

In 0.8, the partitioning key will be stored on the broker. Once kafka-957
is resolved, mirror maker will be able to partition messages based on the
same key in the target cluster.

Thanks,

Jun
On Tue, Jul 2, 2013 at 7:55 PM, 王国栋 <[EMAIL PROTECTED]> wrote:

> HI guys,
>
> We are using kafka0.7.2, in our cluster, we use customized partition
> functions in producer.
> Say, we compute partition id with user id in our log.
>
> But when we use mirror maker to pull log, we find that mirror maker uses
> random partition to push log into destination brokers.
>
> In my mind, we have to decompress logs from consumer and re-partition them
> again. So, I am wondering if there are any good ways to keep partition rule
> in mirror maker.
>
> Thanks,
>
> Guodong
>