Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> how to keep the same partition when using mirror maker


+
王国栋 2013-07-03, 02:56
+
Jun Rao 2013-07-03, 04:54
Copy link to this message
-
Re: how to keep the same partition when using mirror maker
Thanks Jun.

Guodong
On Wed, Jul 3, 2013 at 12:53 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> One of the issues is that in 0.7, the partitioning key is not stored on the
> broker. So, mirror maker won't know what key to use for partitioning.
>
> In 0.8, the partitioning key will be stored on the broker. Once kafka-957
> is resolved, mirror maker will be able to partition messages based on the
> same key in the target cluster.
>
> Thanks,
>
> Jun
>
>
> On Tue, Jul 2, 2013 at 7:55 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
>
> > HI guys,
> >
> > We are using kafka0.7.2, in our cluster, we use customized partition
> > functions in producer.
> > Say, we compute partition id with user id in our log.
> >
> > But when we use mirror maker to pull log, we find that mirror maker uses
> > random partition to push log into destination brokers.
> >
> > In my mind, we have to decompress logs from consumer and re-partition
> them
> > again. So, I am wondering if there are any good ways to keep partition
> rule
> > in mirror maker.
> >
> > Thanks,
> >
> > Guodong
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB