Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Re: partition numbers and mirrormaker


Copy link to this message
-
Re: partition numbers and mirrormaker
Joel Koshy 2013-07-03, 17:27
Libo, if you set num.partitions on the target brokers and set topic
auto creation to true then you don't need to create the topic
beforehand.

On Wed, Jul 3, 2013 at 9:15 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:
> Thanks, Jun. Previously I created the topic with the specified number of partitions on the destination
> (target) cluster before I launched mirrormaker to do mirroring. But the mirrormaker only downloaded
> data for some of partitions, and in some cases, it stopped randomly.  I will test this scenario again with
> beta1.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, July 03, 2013 11:49 AM
> To: [EMAIL PROTECTED]
> Subject: Re: partition numbers and mirrormaker
>
> Yes, but you can change num.partitions in the target brokers.
>
> Thanks,
>
> Jun
>
>
> On Wed, Jul 3, 2013 at 6:21 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:
>
>> For a topic with N partitions on the source cluster, if I let
>> mirrormaker to create topic automatically, the same topic on the
>> destination cluster will have only one partition.
>> Is that expected behavior?  Could you just give a yes/no answer? Thanks.
>>
>> Regards,
>>
>> Libo
>>
>>
>> -----Original Message-----
>> From: Joel Koshy [mailto:[EMAIL PROTECTED]]
>> Sent: Tuesday, July 02, 2013 5:23 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: partition numbers and mirrormaker
>>
>> You can override the default number of partitions on the source and
>> target cluster to N. (num.partitions in the broker config). If you
>> need a higher (or lower) number of partitions for specific topics you
>> can override that on a per-topic basis (again, the config can be kept
>> identical between the source and target cluster).
>>
>> Thanks,
>>
>> Joel
>>
>> On Tue, Jul 2, 2013 at 1:26 PM, Yu, Libo <[EMAIL PROTECTED]> wrote:
>> > Hi
>> >
>> > I asked this question previously and didn't get an answer.
>> > If a topic has N partitions on source cluster, after being mirrored,
>> > on the destination cluster it has only one partition.
>> >
>> > To avoid this issue, before mirroring, I create the same topic with
>> > N partitions on the Destination cluster. But then the mirrormaker
>> > cannot fetch all partitions and stop randomly.
>> >
>> > So what should we expect from mirrormake and what is the right way
>> > to
>> use it? Thanks.
>> >
>> >
>> > Regards,
>> >
>> > Libo
>> >
>>