Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: partition numbers and mirrormaker


Copy link to this message
-
Re: partition numbers and mirrormaker
Libo, if you set num.partitions on the target brokers and set topic
auto creation to true then you don't need to create the topic
beforehand.

On Wed, Jul 3, 2013 at 9:15 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:
> Thanks, Jun. Previously I created the topic with the specified number of partitions on the destination
> (target) cluster before I launched mirrormaker to do mirroring. But the mirrormaker only downloaded
> data for some of partitions, and in some cases, it stopped randomly.  I will test this scenario again with
> beta1.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, July 03, 2013 11:49 AM
> To: [EMAIL PROTECTED]
> Subject: Re: partition numbers and mirrormaker
>
> Yes, but you can change num.partitions in the target brokers.
>
> Thanks,
>
> Jun
>
>
> On Wed, Jul 3, 2013 at 6:21 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:
>
>> For a topic with N partitions on the source cluster, if I let
>> mirrormaker to create topic automatically, the same topic on the
>> destination cluster will have only one partition.
>> Is that expected behavior?  Could you just give a yes/no answer? Thanks.
>>
>> Regards,
>>
>> Libo
>>
>>
>> -----Original Message-----
>> From: Joel Koshy [mailto:[EMAIL PROTECTED]]
>> Sent: Tuesday, July 02, 2013 5:23 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: partition numbers and mirrormaker
>>
>> You can override the default number of partitions on the source and
>> target cluster to N. (num.partitions in the broker config). If you
>> need a higher (or lower) number of partitions for specific topics you
>> can override that on a per-topic basis (again, the config can be kept
>> identical between the source and target cluster).
>>
>> Thanks,
>>
>> Joel
>>
>> On Tue, Jul 2, 2013 at 1:26 PM, Yu, Libo <[EMAIL PROTECTED]> wrote:
>> > Hi
>> >
>> > I asked this question previously and didn't get an answer.
>> > If a topic has N partitions on source cluster, after being mirrored,
>> > on the destination cluster it has only one partition.
>> >
>> > To avoid this issue, before mirroring, I create the same topic with
>> > N partitions on the Destination cluster. But then the mirrormaker
>> > cannot fetch all partitions and stop randomly.
>> >
>> > So what should we expect from mirrormake and what is the right way
>> > to
>> use it? Thanks.
>> >
>> >
>> > Regards,
>> >
>> > Libo
>> >
>>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB