Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - how to keep the same partition when using mirror maker


Copy link to this message
-
Re: how to keep the same partition when using mirror maker
王国栋 2013-07-03, 07:47
Thanks Jun.

Guodong
On Wed, Jul 3, 2013 at 12:53 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> One of the issues is that in 0.7, the partitioning key is not stored on the
> broker. So, mirror maker won't know what key to use for partitioning.
>
> In 0.8, the partitioning key will be stored on the broker. Once kafka-957
> is resolved, mirror maker will be able to partition messages based on the
> same key in the target cluster.
>
> Thanks,
>
> Jun
>
>
> On Tue, Jul 2, 2013 at 7:55 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
>
> > HI guys,
> >
> > We are using kafka0.7.2, in our cluster, we use customized partition
> > functions in producer.
> > Say, we compute partition id with user id in our log.
> >
> > But when we use mirror maker to pull log, we find that mirror maker uses
> > random partition to push log into destination brokers.
> >
> > In my mind, we have to decompress logs from consumer and re-partition
> them
> > again. So, I am wondering if there are any good ways to keep partition
> rule
> > in mirror maker.
> >
> > Thanks,
> >
> > Guodong
> >
>