Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Re: partition numbers and mirrormaker


Copy link to this message
-
Re: partition numbers and mirrormaker
Jun Rao 2013-07-03, 15:49
Yes, but you can change num.partitions in the target brokers.

Thanks,

Jun
On Wed, Jul 3, 2013 at 6:21 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:

> For a topic with N partitions on the source cluster, if I let mirrormaker
> to create topic
> automatically, the same topic on the destination cluster will have only
> one partition.
> Is that expected behavior?  Could you just give a yes/no answer? Thanks.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Joel Koshy [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, July 02, 2013 5:23 PM
> To: [EMAIL PROTECTED]
> Subject: Re: partition numbers and mirrormaker
>
> You can override the default number of partitions on the source and target
> cluster to N. (num.partitions in the broker config). If you need a higher
> (or lower) number of partitions for specific topics you can override that
> on a per-topic basis (again, the config can be kept identical between the
> source and target cluster).
>
> Thanks,
>
> Joel
>
> On Tue, Jul 2, 2013 at 1:26 PM, Yu, Libo <[EMAIL PROTECTED]> wrote:
> > Hi
> >
> > I asked this question previously and didn't get an answer.
> > If a topic has N partitions on source cluster, after being mirrored,
> > on the destination cluster it has only one partition.
> >
> > To avoid this issue, before mirroring, I create the same topic with N
> > partitions on the Destination cluster. But then the mirrormaker cannot
> > fetch all partitions and stop randomly.
> >
> > So what should we expect from mirrormake and what is the right way to
> use it? Thanks.
> >
> >
> > Regards,
> >
> > Libo
> >
>