-Re: partition numbers and mirrormaker
Joel Koshy 2013-07-03, 17:27
Libo, if you set num.partitions on the target brokers and set topic
auto creation to true then you don't need to create the topic
On Wed, Jul 3, 2013 at 9:15 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:
> Thanks, Jun. Previously I created the topic with the specified number of partitions on the destination
> (target) cluster before I launched mirrormaker to do mirroring. But the mirrormaker only downloaded
> data for some of partitions, and in some cases, it stopped randomly. I will test this scenario again with
> -----Original Message-----
> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, July 03, 2013 11:49 AM
> To: [EMAIL PROTECTED]
> Subject: Re: partition numbers and mirrormaker
> Yes, but you can change num.partitions in the target brokers.
> On Wed, Jul 3, 2013 at 6:21 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:
>> For a topic with N partitions on the source cluster, if I let
>> mirrormaker to create topic automatically, the same topic on the
>> destination cluster will have only one partition.
>> Is that expected behavior? Could you just give a yes/no answer? Thanks.
>> -----Original Message-----
>> From: Joel Koshy [mailto:[EMAIL PROTECTED]]
>> Sent: Tuesday, July 02, 2013 5:23 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: partition numbers and mirrormaker
>> You can override the default number of partitions on the source and
>> target cluster to N. (num.partitions in the broker config). If you
>> need a higher (or lower) number of partitions for specific topics you
>> can override that on a per-topic basis (again, the config can be kept
>> identical between the source and target cluster).
>> On Tue, Jul 2, 2013 at 1:26 PM, Yu, Libo <[EMAIL PROTECTED]> wrote:
>> > Hi
>> > I asked this question previously and didn't get an answer.
>> > If a topic has N partitions on source cluster, after being mirrored,
>> > on the destination cluster it has only one partition.
>> > To avoid this issue, before mirroring, I create the same topic with
>> > N partitions on the Destination cluster. But then the mirrormaker
>> > cannot fetch all partitions and stop randomly.
>> > So what should we expect from mirrormake and what is the right way
>> > to
>> use it? Thanks.
>> > Regards,
>> > Libo