Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - how to keep the same partition when using mirror maker


Copy link to this message
-
how to keep the same partition when using mirror maker
王国栋 2013-07-03, 02:56
HI guys,

We are using kafka0.7.2, in our cluster, we use customized partition
functions in producer.
Say, we compute partition id with user id in our log.

But when we use mirror maker to pull log, we find that mirror maker uses
random partition to push log into destination brokers.

In my mind, we have to decompress logs from consumer and re-partition them
again. So, I am wondering if there are any good ways to keep partition rule
in mirror maker.

Thanks,

Guodong

 
+
Jun Rao 2013-07-03, 04:54
+
王国栋 2013-07-03, 07:47